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OPERATIONAL  TEST  AND  EVALUATION 


This  instruction  provides  guidance  and  procedures  for  operational  test  and  evaluation  (OT&E)  in  the  Air  Force.  It  applies  to 
all  agencies  involved  in  or  supporting  OT&E.  It  describes  how  to  prepare,  plan  for,  and  report  on  operational  test.  It 
implements  the  OT&E  policies  outlined  in  Department  of  Defense  (DoD)  Directive  5000.1,  23  February  1991;  DoD 
Instruction  5000.2,  23  February  1991;  DoD  5000.2-M,  23  February  1991;  and  AFPD  99-1,  To  ensure  a  full  understanding  of 
the  test  process  this  instruction  should  be  suplemented  by  Air  Force  Instruction  (AFI)  99-103.  In  addition,  users  of  this 
instruction  should  familiarize  themselves  with  the  referenced  DoD  guidance,  Air  Force  63-  series  publications,  and  other  AFI 
99-series  publications.  Implementing,  operating,  and  supporting  commands  or  agencies,  within  the  Air  Force,  may 
supplement  this  instruction.  Send  draft  or  proposed  Major  Command  (MAJCOM)  OT&E  instructions  and  supplements  to 
HQ  AFOTEC/XR.  Send  final  MAJCOM  OT&E  instructions  and  supplements  to  HQ  USAF  /TEP  and  HQ  AFOTEC/XR. 
Submit  recommended  changes  to  this  instruction  via  AF  Form  847,  Recommendation  for  Change  of  Publication,  to  HQ 
AFOTEC/XR  with  info  copy  to  HQ  USAF/TEP. 
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Chapter  1 

GENERAL  POLICY  AND  GUIDANCE 

1.1.  Test  and  Evaluation  (T&E)  Purpose.  The  Air 
Force  needs  to  ensure  the  weapon  systems  it  uses,  and  the 
systems  that  support  them,  meet  or  exceed  operational 
requirements  in  terms  of  effectiveness  and  suitability  and 


are  ready  for  fielding.  To  do  so,  the  Air  Force  conducts 
realistic,  cost-effective,  and  credible  T&E  programs 
through  all  phases  of  the  acquisition  and  fielding  process. 
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1.2.  Operational  Test  and  Evaluation  Purpose.  The 
primary  purpose  of  OT&E  is  to  determine  the  operational 
effectiveness  and  suitability  of  systems.  Testers  conduct 
OT&E  in  as  realistic  an  operational  environment  as 
possible  to  determine  if  a  system  meets  the  users’ 
requirements  and  supports  mission  accomplishment. 
These  realistic  conditions  should  be  representative  of  both 
combat  stress  and  peacetime  operational  conditions. 
Testers  use  modeling  and  simulation  (M&S)  as  an 
evaluation  tool  to  augment,  extend,  or  enhance  field  test 
results.  Testers  conduct  OT&E  to: 

•  Determine  the  ability  of  a  system  to  support 
operational  task  accomplishment 

•  Answer  critical  operational  issues  (COI)  and 
determine  if  the  key  parameters  are  met 

•  Identify  and  report  operational  capabilities, 
limitations,  and  deficiencies  to  include: 

•  Documenting  deficiencies  for  resolution 

•  Recommending  and  evaluating  changes 
in  system  configuration 

•  Providing  information  for  developing 

and  refining  logistics  and  software 
support  requirements  for  the  system 

•  Providing  information  for  developing 

and  refining  training,  tactics, 
techniques,  and  doctrine  throughout  the 
life  of  the  system 

•  Providing  information  for  developing 

and  refining  environmental  impact 
statements  (EIS) 

•  Provide  information  for  refining  operation  and 
maintenance  (O&M)  cost  estimates,  or  identify 
system  characteristics  or  deficiencies  that  affect 
O&M  costs 

•  Determine  if  support  equipment  and  technical 
publications  support  the  mission 

•  Assess  the  survivability  of  the  system  in  the 
operational  environment 

•  Assess  interoperability  and  provide  inputs  to  the 
Joint  Interoperability  Test  Center  (JITC)  for  joint 
interoperability  certification 

1.3.  Types  of  OT&E.  There  are  three  basic  types  of 
OT&E. 


1.3.1.  Initial  Operational  Test  and  Evaluation 
(lOT&E).  lOT&E  is  required  to  be  done  on  production 
or  production  representative  articles  to  support  a 
Milestone  III  full-rate  production  decision.  It  is 
conducted  by  the  Air  Force  Operational  Test  and 
Evaluation  Center  (AFOTEC)  under  conditions  that  are  as 
operationally  realistic  as  possible  and  practical.  AFOTEC 
plans  and  conducts  all  Air  Force  lOT&E  using  3600 
(Research  and  Development  (R&D))  funds.  lOT&E  can 
also  be  used  to  support  fielding  or  initial  operational 
capability  (IOC)  decisions  as  well  as  assist  in  tactics 
development  for  the  using  command. 

1.3.2.  Qualification  Operational  Test  and  Evaluation 
(QOT&E).  QOT&E  is  identical  to  lOT&E,  in  that 
QOT&E  is  accomplished  on  production  representative 
articles,  under  conditions  that  are  as  operationally  realistic 
as  possible  and  practical.  AFOTEC  will  conduct  QOT&E 
on  programs  where  there  has  been  no  R&D  effort  such  as 
non-developmental  items  (NDI),  commercial  off-the-shelf 
(COTS)  systems,  or  certain  modifications  to  existing 
systems.  QOT&E  is  conducted  by  AFOTEC  using  3400 
(O&M)  funds  and  the  results  support  Milestone  III 
procurement,  fielding,  or  IOC  decisions. 

1.3.3.  Follow-on  Operational  Test  and  Evaluation 
(FOT&E).  FOT&E  is  conducted  over  the  life  of  the 
system  to  ensure  the  system  continues  to  meet  user 
requirements  and  to  explore  non-material  means  of 
satisfying  deficiencies.  Air  Force  MAJCOM  are 
responsible  for  the  conduct  of  FOT&E.  In  certain  cases, 
HQ  USAF  will  direct  AFOTEC  to  conduct  a  specific 
FOT&E.  The  Air  Force  conducts  FOT&E  to: 

•  Refine  estimates  made  during  lOT&E  or 
QOT&E 

•  Complete  deferred  lOT&E  or  QOT&E  testing 

•  Evaluate  changes  and  verify  correction  of 
deficiencies 

•  Assist  in  tactics  development 

•  Reevaluate  the  system  to  ensure  it  continues  to 
meet  operational  needs 

The  Air  Force  uses  3400  (O&M)  funds  for  FOT&E. 

1.4.  Combining  OT&E  with  DT&E.  OT&E  and 
developmental  test  and  evaluation  (DT&E)  are  separate 
and  distinct,  but  the  resources  needed  to  conduct  and 
support  the  two  efforts  are  often  similar.  Much  of  the  test 
data  generated  by  each  are  beneficial  to  both;  therefore, 
portions  of  DT&E  and  OT&E  will  be  combined  when 
prudent  to  expedite  system  acquisition  and  reduce  costs. 
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Single  managers  (SM)  will  document  combined  tests  in 
the  Test  and  Evaluation  Master  Plan  (TEMP).  When 
planning  combined  testing,  developmental  and 
operational  testers  must  integrate  the  necessary  test 
conditions  and  data  requirements.  Combined  testing  will 
not  compromise  the  objectives  of  DT&E  or  OT&E. 
Operational  testers  must  conduct  a  dedicated  phase 
(portion  entirely  independent  of  the  developer,  contractor, 
and  user)  prior  to  the  production/procurement  decision 
(Milestone  III).  Refer  to  Air  Force  Instruction  (AH)  99- 
101,  Developmental  Test  and  Evaluation,  for  additional 
guidance  and  procedures  on  DT&E  and  combined  testing. 

1.5.  Conducting  OT&E  with  other  services/agencies 
(Multiservice  OT&E).  Multiservice  OT&E  is  conducted 
with  other  government  agencies  or  other  service 
operational  test  agency  (OTA)  participation.  In  general, 
multiservice  OT&E  is  done  to  support  multiservice 
acquisitions.  Multiservice  OT&E  will  be  conducted 
according  to  the  T&E  directives  of  the  lead  test 
service/agency,  or  as  agreed  to  between  participants  in  a 
memorandum  of  agreement  (MOA).  The  lead 
service/agency  will  include  supporting  service  test 
requirements  in  the  multiservice  test  plans.  When  the  Air 
Force  is  lead  service,  the  Air  Force  will  attempt  to 
incorporate  all  the  services’  unique  requirements  into  a 
single  plan.  If  separate  service  plans  are  required  they 
will  be  included  as  attachments  to  the  Air  Force  plan.  The 
Air  Force  will  publish  a  single  report  with  the  other 
services’  results  attached. 

1.6.  Other  OT&E-Related  Activities. 

1.6.1.  Operational  Utility  Evaluation  (OUE).  OUEs 
pertain  to  those  operational  tests  clearly  outside  the  scope 
of  the  tests  described  in  paragraph  1.3.  OUEs  are  not 
limited  to,  but  may  be  conducted  to  validate  a  concept  or 
expand  the  mission  of  an  existing  (perhaps  modified) 
weapon  system  to  a  different  role  or  mission.  OUEs  are 
typically  HQ  USAF-directed  and  AFOTEC  or  MAJCOM- 
conducted  and  are  specifically  limited  in  time  and  scope. 
OUEs  are  funded  with  the  same  type  of  appropriations  as 
the  parent  program. 

1.6.2.  Operational  Assessments  (OA).  OAs  are 
normally  precursors  to  lOT&E/QOT&E  conducted  by 
AFOTEC  to  provide  information  to  developers,  users,  and 
decision  makers.  AFOTEC  will  not  conduct  OAs  as  a 
substitute  for  or  in  lieu  of  OT&E.  OAs  conducted  prior  to 
MS  II  are  early  operational  assessments  (EOA). 
AFOTEC  will  conduct  OAs  as  directed  by  or  coordinated 
with  AF/TE.  OAs  provide  the  following  kinds  of 
information: 

•  Assessments  of  major  impacts  to  operational 
effectiveness  and  suitability 


•  Identification  of  programmatic  voids  adversely 
impacting  the  ability  to  meet  operational 
requirements 

•  Testability  of  operational  requirements 

•  Ability  of  programs  to  support  operational 
testing 

•  Readiness  of  a  system  for  low-rate  initial 
production  (LRIP) 

•  Answers  to  specific  questions  or  issues  as  raised 
by  senior  decision  makers 

1.6.3.  Tactics  Development  and  Evaluation  (TD&E), 
TD&E  is  the  formal  portion  of  a  Combat  Air  Forces 
tactics  development  program  designed  to  fully  exploit  a 
system’s  capabilities.  It  includes  the  research,  analysis, 
development,  test,  and  evaluation  of  specific  employment 
tactics  against  anticipated  threats.  The  using  command 
conducts  TD&E,  using  the  same  standards  and  rigor  as 
OT&E. 


1.6.4,  Joint  Test  and  Evaluation  (JT&E).  JT&E  is 
used  to  evaluate  technical  or  operational  concepts  which 
have  applicability  to  more  than  one  service.  JT&Es  are 
Office  of  the  Secretary  of  Defense  (OSD)-directed  and 
funded.  Refer  to  AFI  99-106,  Joint  Test  and  Evaluation, 
for  more  information. 


1.7.  Waiver  Requests.  The  HQ  USAF,  Director,  Test 
and  Evaluation  (HQ  USAF/TE)  is  the  sole  waiver 
authority  for  this  instruction.  Submit  waiver  requests  in 
writing  through  HQ  AFOTEC/CC  to  HQ  USAF/TE. 
Waiver  requests  with  rationale  must  be  submitted  with 
user,  developer,  and  tester  concurrence.  Once  HQ 
USAF/TE  approves  waivers  in  writing,  single  managers 
in  the  TEMP  and  the  program  officers  in  the  Program 
Management  Directive  (PMD)  will  document  them. 

1.8.  Management  of  OT&E, 

1,8.1.  Office  of  the  Secretary  of  Defense,  Director, 
Operational  Test  and  Evaluation  (OSD/DOT&E): 

•  Reports  directly  to  the  Secretary  of  Defense 
(SECDEF)  and  Congress 

•  Formulates  OT&E  policy  and  procedures  for 
DoD  and  is  the  principal  advisor  to  SECDEF  for 
OT&E  matters 
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•  Exercises  oversight  responsibility  over  major 

defense  acquisition  programs  or  any  program  in 
which  the  SECDEF,  OSD,  or  Congress  has 
special  interest  as  defined  in  an  annually 

published  OSD  T&E  oversight  list 

•  Approves  the  TEMP  in  conjunction 
with  Director,  Test  and  Evaluation 
(OSD/DT&E) 

•  Approves  the  adequacy  of  the  OT&E 
plan 

•  Determines  the  quantity  of  operational 
test  articles  for  Acquisition  Category 
(ACAT)  I  programs 

•  Submits  a  beyond  low-rate  initial 
production  (B-LRIP)  report  to  the 
SECDEF,  the  Under  Secretary  of 
Defense  for  Acquisition  and 
Technology  (USD(A&T)),  and  the 
congressional  committees  on  Armed 
Services  and  Appropriations  before  the 
system  can  be  committed  to  full-rate 
production 

1.8.2.  HQUSAF/TE; 

•  Is  responsible  for  policy,  advocacy  of  test 
resources,  and  oversight  of  all  Air  Force  T&E 
processes  and  programs 

•  Reviews  and  coordinates  on  Mission  Need 
Statements  (MNS),  Cost  and  Operational 
Effectiveness  Analyses  (COEA),  Operational 
Requirements  Documents  (ORD),  and  other 
T&E-related  documentation  used  by  the  Air 
Staff,  DoD,  and  the  Congress 

•  Determines,  in  conjunction  with  AFOTEC,  the 
OT&E  strategy  in  support  of  a  system’s 
acquisition  strategy  and  designates  test 
responsibilities  in  the  PMD  and  TEMP 

•  Acts  as  the  final  review  authority  and  signs 
TEMPs  prior  to  Air  Force  Acquisition  Executive 
(AFAE)  approval 

•  Reviews  and  authorizes  the  release  of  any  Air 
Force  test  concept  briefing  and  OT&E  plan 
outside  the  Air  Force 

1.8.3.  Assistant  Secretary  of  the  Air  Force  for 

Acquisition  (SAF/AQ): 


•  Designated  the  Service  Acquisition  Executive 

•  Documents  system  T&E  responsibilities  through 
the  PMD 

•  Acts  as  the  Air  Force  TEMP  approval  authority 
for  ACAT  I  and  II  and  all  selected  OSD  T&E 
oversight  program 

•  Ensures  Program  Executive  Officers  (PEO)  and 
Designated  Acquisition  Commanders  (DAC) 
certify  program  readiness  for  dedicated  OT&E 

1.8.4.  AFOTEC: 

•  Reports  directly  to  the  Air  Force  Chief  of  Staff 
(CSAF) 

•  Manages  the  Air  Force  OT&E  program 

•  Determines  the  number  of  operational  test 
articles  for  ACAT  II-IV,  I/QOT&E  programs 

•  Plans,  conducts,  and  reports  all  lOT&E  and 
QOT&E 

•  Conducts  HQ  USAF/TE-directed  FOT&E,  OAs, 
and  OUEs 

1.8.5.  Single  Manager  (SM).  The  SM  may  be  a  system 
program  director  (SPD),  product  group  manager,  or 
materiel  group  manager. 

•  Manages  all  aspects  of  a  specific  acquisition 
program 

•  Responsible  for  establishing  the  Test  Plan 
Working  Group  (TPWG) 

•  Develops  and  coordinates  the  TEMP,  in 
conjunction  with  the  Responsible  Test 
organization  (RTO)  and  operational  test  agency 

•  Provides  the  required  number  of  test  articles  as 
determined  by  OSD  or  AFOTEC 

•  Provides  acquisition  program  strategy 
documentation  to  AFOTEC 

•  Establishes  and  manages  the  deficiency  reporting 
process 

1.8.6.  Responsible  Test  Organization: 

•  Conducts  and  reports  on  DT&E  in  support  of  the 
acquisition  program  (See  AH  99-101) 
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•  Integrates  DT&E  and  OT&E  test  requirements 
for  combined  T&E  programs 

•  Participates  in  the  certification  of  readiness  for 
dedicated  OT&E 

1.8.7,  MAJCOMs: 


Plan,  conduct, 

•  Are  responsible  for  the  MNS,  COEA,  ORD,  and 
Concept  of  Operations  (CONOPS) 

•  Plan,  conduct,  and  report  on  FOT&E,  TD&E, 
and  other  testing  designated  by  HQ  USAF 


Chapter  2 

ACQUISITION  AND  TEST  DOCUMENTATION 


2.1.  Introduction.  The  operational  tester  must  be 
familiar  with  key  acquisition  and  test  documents  and  their 
relationships  to  one  another  in  order  to  plan  effective 
OT&E.  Key  acquisition  and  test  documents  are  listed. 
Attachment  1  provides  a  more  complete  listing  of  test  and 
test-related  documentation  related  to  OT&E. 

2.2.  Key  Documents  and  Their  Relevance  to  the 
OT&E  Process.  Test  planners  must  ensure  clear  and 
consistent  links  exist  between  the  system  operational 
requirements  documents,  program  acquisition  documents, 
and  the  planning,  conduct,  execution,  and  reporting  of 
OT&E. 

2.2.1.  Mission  Need  Statement  (MNS).  Users 
(MAJCOMs,  Commanders  in  Chief,  or  HQ  USAF) 
prepare  a  MNS  upon  identification  of  a  deficiency  to 
accomplish  a  task  or  mission,  that  cannot  be  satisfied  by  a 
change  in  tactics,  doctrine,  or  training.  The  MNS 
identifies  and  documents  mission  deficiencies  that  require 
materiel  solutions  in  generic,  non -system  specific, 
operational  terms.  An  approved  MNS  (see  AFI  10-601, 
Mission  Needs  and  Operational  Requirements  Guidance 
and  Procedures,  for  approval  process)  provides  the  initial 
input  for  identification  of  T&E  critical  operational  issues. 

2.2.2.  Acquisition  Decision  Memorandum  (ADM). 
After  approval  of  the  MNS,  issuance  of  an  ADM,  directs 
concept  studies  of  alternative  programs  (Concept 
Exploration  and  Definition  Phase).  The  ADM 
summarizes  the  USD(A)’s,  or  designated  representative’s, 
decisions  and  exit  criteria  established  during  the 
milestone  decision  review  or  in-process  review.  The 
ADM  contains  directions  for  acquisition  personnel  to 
follow  through  the  next  acquisition  phase.  Operational 
testers  must  be  aware  of  the  decisions  documented  in  the 
ADM  and  the  impacts  to  the  test  program.  The 
operational  tester  must  evaluate  the  testability  and  their 
implications  for  each  of  the  concepts  being  studied.  Refer 
to  AF  Sup  1  to  DoDI  5000.2,  Acquisition  Management 
Policies  and  Procedures. 


2.2.3.  Program  Management  Directive  (PMD).  HQ 
USAF  prepares  a  draft  PMD  to  initiate  concept  studies 
after  the  Milestone  Decision  Authority  (MDA)  authorizes 
the  Concept  Exploration  and  Definition  Phase  (Phase  0) 
in  the  ADM.  The  PMD  provides  program  direction  and 
guidance  to  commands.  The  PMD  designates 
implementing,  participating,  supporting,  and  OT&E 
commands,  their  program  responsibilities,  and 
relationships.  The  PMD  also  lists  review  and  approval 
requirements,  program  objectives,  constraints,  funding, 
the  decision  authority,  and  identifies  the  progression 
prerequisites  for  the  next  acquisition  phase.  Refer  to  HQ 
USAF  Headquarters  Operating  Instruction  (HOI)  800-2, 
Policy  and  Guidance  for  Preparing  Program 
Management  Directives. 

2.2.4.  Cost  and  Operational  Effectiveness  Analysis 
(COEA).  The  COEA  provides  an  analytical  basis  to 
support  acquisition  milestone  decision  reviews.  The 
COEA  compares  alternative  solutions  on  the  basis  of  cost 
and  operational  effectiveness,  documents  analytical 
rationale  for  preferring  one  alternative  over  another, 
justifies  the  need  for  starting  or  continuing  an  acquisition 
program,  and  effectively  communicates  the  results  to  all 
levels  of  Air  Force  leadership.  The  COEA  provides  early 
identification  of  potential  OT&E  measures  of 
effectiveness  (MOE)  and  measures  of  performance 
(MOP).  Operational  test  agencies  must  be  involved  in  the 
development  of  COEA  MOEs  and  MOPs,  ensuring  the 
measures  can  either  be  directly  tested,  evaluated,  or 
derived  from  other  sources  such  as  M&S.  Refer  to  AFI 
10-601. 

2.2.5.  Operational  Requirements  Document  (ORD). 
The  PMD  directs  the  user  to  develop  an  ORD  according 
to  DoDI  5000.2  and  AFI  10-601.  The  ORD  is  solution 
oriented  and  will  be  based  on  the  preferred  solution 
selected  by  the  MAJCOM  Commander  in  the  COEA. 
ORDs  contain  essential  quantitative  and  qualitative 
operational  requirements  for  the  proposed  system  and  are 
the  key  to  understanding  user  priorities.  Testers  must 
review  draft  ORDs  carefully  in  order  to  structure  test 
concepts  and  planning.  The  ORD  also  provides 
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preliminary  tactics  and  doctrine  information,  projected 
climate,  terrain,  threats,  and  training  and  maintenance 
concepts.  These  inputs  define  the  projected  mission  and 
are  essential  to  planning  operationally  realistic  tests. 
Operational  testers  do  not  develop  requirements,  but 
ensure  user-provided  requirements  can  be  measured  and 
evaluated. 

2.2.6.  System  Threat  Assessment  Report  (STAR).  The 
STAR,  for  AC  AT  I  programs,  or  the  System  Threat 
Assessment  (STA)  for  ACAT  II  programs,  is  the  primary 
source  for  system  specific  threat  information.  The  MNS, 
ORD,  and  threat  environment  description  (TED)  provide 
general  threat  information.  The  TED  contains  a 
comprehensive  description  of  threats,  including  references 
to  other  intelligence  data.  Early  in  the  test  planning 
process,  OTAs  will  prepare  a  threat  summary  based  on 
the  STAR  or  STA,  related  intelligence  documents,  and 
user  requirements.  This  threat  summary  will  be  a  living 
document  evolving  in  detail  as  the  operational  concept 
and  system  design  mature.  Threat  summaries  will  contain 
a  comprehensive  unconstrained  view  of  the  operational 
environment  (IOC  to  IOC  plus  10  years)  and  should  be 
the  basis  for  creating  a  realistic  operational  test 
environment.  Testers  will  consider  detailed  threats  a 
system  will  encounter  when  developing  the  test  strategy. 
Operational  testers  must  include  the  threat  summary  as  a 
supplement  to  the  OT&E  plan.  Refer  to  API  14-201, 
Intelligence  Production  and  Applications  Requirements 
for  additional  information. 

2.2.7.  Test  and  Evaluation  Master  Plan  (TEMP).  The 
TEMP  establishes  a  comprehensive  strategy  for  planning 
and  executing  a  system’s  T&E  program.  The  TEMP 
includes  the  operational  tasks,  characteristics,  and 
capabilities  (as  described  in  the  MNS  and  ORD).  The 
TEMP  documents  how  testers  will  address  the  MOEs, 
MOPs,  and  associated  test  resources.  Operational  testers 
will  include,  in  the  TEMP,  a  description  of  any  M&S  (to 
include  accreditation  approach)  they  will  use  in  OT&E. 
Refer  to  DoDI  5000.2;  DoD  5000. 2-M,  Defense 
Acquisition  Management  Documentation  and  Reports',  AF 
Sup  1  to  DoD  5000.2M;  and  AH  99-101.  All  ACAT  I 
through  IV  and  PMD-directed  programs  will  have  a 
TEMP.  The  TEMP  accomplishes  three  things: 

Note:  Testers  should  submit  shortfalls  as  resource 

requirements  to  the  test  investment  planning  and 
programming  (TIPP)  process.  For  a  description  of  the 


TIPP  process,  refer  to  AFI  99-109,  Test  Resource 
Planning. 

•  Functions  as  the  primary  document  through 
which  the  T&E  program  is  reviewed  and/or 
approved  by  SAF/AQ,  HQ  USAF,  OSD,  and 
Congress. 

2.2.7. 1.  Test  Agencies  Responsibilities.  Test  agencies 
assist  the  SM  in  integrating  ’’user  defined”  critical 
operational  issues  and  system  characteristics;  test 
objectives  and  evaluation  criteria;  and  OT&E 
responsibilities,  resources,  and  schedule  into  the  TEMP. 
Test  agencies  will  ensure  system  characteristics, 
capabilities,  COIs,  critical  parameters,  and  their  criteria, 
documented  in  the  TEMP,  are  directly  traceable  to  the 
ORD. 

2.3.  Other  Supporting  Documentation. 

2.3.1.  Program  Introduction  Document  (PI,  also  PID). 
Operational  test  agencies  normally  prepare  a  PI  for  most 
test  programs  having  range  requirements.  Pis  are 
submitted  to  DoD  major  range  and  test  facilities  as 
official  test  program  notification. 

2.3.2.  Support  Agreements.  Support  agreements 
delineate  general  host  base  support  requirements  and 
tenant  test  team  and  test  organization  responsibilities. 
Testers  must  address  in  support  agreements: 

•  Unique  safety  support  requirements 

•  Handling  and  disposal  of  any  hazardous 
materials 

•  Safety  and  resource  protection  (to  include 
weather  warning  and  advisories) 

Refer  to  AFI  25-201,  Support  Agreements  Requirements, 

2.3.3.  Memorandum/Letter  of  Agreement  or 
Understanding  (MOA,  MOU,  LOA,  LOU).  When  an 
interservice  support  agreement  or  a  PI  can  not  satisfy 
tester  requirements,  an  MOA/LOA  or  MOU/LOU  should 
be  established  between  concerned  agencies  to  formally 
document  respective  tasks  and  responsibilities.  Refer  to 
AFI  25-201. 
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Chapter  3 

OT&E  ACTIVITIES 


3.1.  OT&E  in  the  Acquisition  Process.  The  DoD 
system  acquisition  process  is  divided  into  five  phases 
(Figure  1).  During  Phase  0  and  Phase  I,  OT&E  assists  in 
development  of  the  acquisition  strategy  and  MOEs,  and 
develops  the  OT&E  concept.  OT&E  involvement  during 


Phase  II  includes  the  planning  and  execution  of 
lOT&E/QOT&E.  During  Phases  III  and  IV,  FOT&E  is 
conducted  to  ensure  a  system  retains  its  effectiveness  in  a 
new  environment  or  against  new  or  changing  threats. 


Figure  1.1.  The  Acquisition  Process. 
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3.1.1.  Tailoring  OT&E.  Some  acquisition  programs 
may  not  follow  the  standard  system  acquisition  process. 
Examples  of  programs  that  may  require  tailoring  are: 

•  Systems  where  the  decision  to  buy  must  precede 
the  completion  of  OT&E  as  in  one-of-a-kind 
systems 

•  Systems  which  require  early  testing  on  non¬ 
production  representative  articles  or  prototypes 

•  Systems  where  there  is  limited  or  non 
availability  of  "hands-on"  test  capability 

•  Systems  that  must  maintain  an  on-line 

operational  capability 

•  Systems  operated  or  maintained  by  contractors 

•  Systems  which  use  modeling  and  simulation  to 
fill  voids  caused  by  cost  or  availability  of  test 
resources 

•  Advanced  Concept  Technology  Demonstration 
(ACTD)  programs 

•  "Fast  Track"  Acquisition  systems  -  Systems  built 
to  a  specific  combat  mission  need,  requiring 
fielding  in  minimum  time  to  support 
requirements  during  combat 

•  Evolutionary  Acquisition  System  and 
Modifications  -  Characterized  by  the  design, 
development,  and  deployment  of  a  preliminary 
capability  that  includes  provisions  for  the 
evolutionary  addition  of  future  functionality  and 
changes,  as  requirements  are  further  defined 

•  Incremental  Acquisition  Systems  -  Characterized 
by  the  development,  acquisition,  and  deployment 
of  functionality  through  a  number  of  clearly 
defined  system  "increments"  that  stand  on  their 
own 

3.1.2.  Though  the  following  guidance  is  built  on  the 
framework  of  a  major  defense  acquisition  program,  the 
philosophy  presented  should  be  applied  to  all  OT&E  and 
ACAT  programs.  Because  each  acquisition  program  is 
unique,  the  operational  test  community  must  adapt  to 
support  the  acquisition  strategy  being  employed.  When 
adapting  the  test  strategy,  the  test  community  must  still 
ensure  system  effectiveness  and  suitability.  This  can  be 
done  by  lOT&Es,  QOT&Es,  FOT&Es,  and  possibly  OAs, 
or  OUEs  or  any  combination  of  tests.  It  is  the 
responsibility  of  the  operational  test  agency  and  AF/TE  to 
determine  the  most  cost-effective  and  efficient  manner  to 


test  while  still  ensuring  a  correct  evaluation  of  system 

effectiveness  and  suitability. 

3.2.  Pre-Milestone  0. 

•  The  conduct  of  FOT&E  may  identity  shortfalls 
and  deficiencies,  which  can  be  used  by  the 
MAJCOM  to  form  the  basis  of  a  MNS  prior  to 
MS  0. 

•  AFOTEC  begins  program  monitoring  with  the 
issuance  of  a  validated  MNS  documenting  a 
command’s  mission  need. 

3.3.  Milestone  0  -  Concept  Studies  Approval. 

•  At  MS  0,  senior  leadership  determine  if  the 
documented  mission  need  warrants  the  initiation 
of  study  efforts  of  alternative  concepts. 

•  Following  MS  0,  AFOTEC  will  be  involved  in 
the: 

•  Development  of  a  COEA 

•  Development  of  an  ORD 

•  Development  of  an  initial  TEMP  (based  on  the 
evolving  preferred  solution) 

•  Development  of  other  program  documentation  to 
support  a  Milestone  I  decision 

3.4.  Phase  0  -  Concept  Exploration  and  Definition. 

•  MAJCOM  FOT&E  may  continue  and  be  used  to 
help  explore  the  feasibility  of  proposed  concepts 

•  Early  AFOTEC  involvement  during  this  phase 
will  involve  participation  in  the  COEA,  ORD, 
and  Temp  development. 

3.4.1.  During  COEA  development,  AFOTEC  will: 

•  Analyze  concepts  being  explored  and  assess  the 
options  under  consideration  for  feasibility  of 
testing  and  evaluation 

•  Identify  as  early  as  possible  shortfalls  in  the 
ability  to  test  proposed  concepts  and  preferred 
solutions;  offer  fixes  or  alternatives 

•  Work  MOE/MOP  issues  with  the  users  for  the 
proposed  preferred  solution 
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3.4.2.  During  OBJ)  development  (Refer  to  AFI  10-601 
for  ORD  development),  AFOTEC  will: 

•  Review  the  draft  ORD  for  early  identification  of 
measurable  and  testable  requirements  written  in 
operational  terms  (not  specifications).  Ensure 
sufficient  operations  concept  and  employment 
descriptions  are  specified  to  develop  operational 
test  scenarios.  Work  with  the  user  to  get  these 
changes  in  the  final  MS  I  ORD 

3.4.3.  During  TEMP  preparation  (Refer  to  AF  Sup  1  to 
DODI  5000.2),  AFOTEC  will: 

•  Create  and  document  an  OT&E  strategy  to 
support  the  acquisition  of  the  preferred  solution 

•  Interface  the  OT&E  strategy  with  the 
acquisition  strategy  and  other  test  strategies  (e.g. 
DT&E,  Contractor  Test  (CT),  Live  Fire  Test, 
etc.) 

•  Work  test  planning  issues  with  HQ 
AFMC  or  PM/SPO  until  a  TPWG  is  formed. 
(Refer  to  AH  99-101  for  TPWG) 

•  Prepare  preliminary  TEMP  inputs 

•  Provide  recommended  OT&E  strategy  to  AF/TE 
for  approval. 

3.5.  Milestone  I  -  Concept  Demonstration 
Approval.  Senior  leadership  evaluate  the 
program  and  determine  if  the  results  of  Phase  0 
warrant  establishing  a  new  acquisition  program 
and  which  option  will  be  pursued.  In  addition, 
initial  TEMP  approval  occurs  at  MS  I. 

3.6.  Phase  I  -  Demonstration  and  Validation. 

During  Phase  I,  SAF/AQ  issues  a  PMD  directing 
the  program.  In  addition,  the  user  continues  to 
refine  the  COEA  and  the  ORD.  As  data 
becomes  available,  the  ORD  is  refined. 

3.6.1.  During  Phase  I,  AFOTEC  will: 

•  Work  with  the  user  to  review  and  refine  the 
COEA  and  ORD 

•  Continue  to  develop  and  refine  the  OT&E 
strategy,  outlining  the  plan  and  determining 
sequence  of  data  needed  by  decision  makers  (i.e., 
OUE,  OA,  lOT&E,  FOT&E) 

•  Continue  to  review  and  refine  OT&E  portions  of 
the  TEMP  through  the  TPWG  process 


•  Conduct  over-the-shoulder  observation/ 
assessment  of  ongoing  DT&E  and  CT,  where 
appropriate 

•  Participate  in  the  planning  and  conduct  of 
combined  testing,  when  combined  testing  is 
directed 

•  Plan  and  conduct  an  EOA,  when  directed,  for 
MS  II  (LRIP  and  Long-lead  decisions) 

•  Determine  LRIP  numbers  for  OT&E  on  ACAT 
II  programs 

3.6.2.  During  Phase  I,  AF/TE  will  approve,  modify,  or 
direct  changes  to  the  OT&E  strategy,  as  necessary,  via  the 
TPWG  and  SAF/AQ  so  changes  can  be  incorporated  in 
the  TEMP  and  PMD  updates. 

3.7.  Milestone  11  Decision  -  Development  Approval. 
Senior  Leadership  evaluate  the  program  and  decide  if  the 
program  warrants  continuation  into  Phase  II,  Engineering 
and  Manufacturing  Development.  In  addition,  approval 
for  low-rate  initial  production  quantities,  if  appropriate, 
occurs. 

3.8.  Phase  II  -  Engineering  and  Manufacturing 
Development.  During  Phase  II,  AFOTEC  will: 

•  Continue  to  review  and  refine  OT&E  portions  of 
the  TEMP  through  the  TPWG  process 

•  Continue  development  and  brief  the  OT&E  test 
concept  (See  Paragraph  4. 1 ) 

•  Conduct  over-the-shoulder  observation  of 
ongoing  DT&E  and  CT,  where  appropriate 

•  Ensure  ORD  is  updated  to  reflect  the  most 
current  user  requirements  and  that  clear,  testable 
MOE  criteria  are  established 

•  Conduct  OAs  as  directed  by  HQ  USAF/TE 

•  Support  LRIP  decisions  if  made  post-MS  II 

•  Participate  in  Combined  Testing,  when  it  is 
directed 

•  Develop  the  I/QOT&E  plan.  (See  paragraph  4.2 
and  Attachment  2) 

•  Participate  in  and  support  the  PEO/DAC’s 
Certification  of  Readiness  for  Dedicated  lOT&E 
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•  Ensure  the  TEMP,  and  OT&E  plan  reflect  the 
most  recent  ORD  requirements  before  entering 
test 

•  Conduct  I/QOT&E  according  to  that  plan 

•  Report  (via  briefings  and  a  test  report)  lOT&E 
results  to  support  a  MS  III  decision  (See 
paragraph  4.4  and  Attachment  3) 

•  Participate  in  Pre-Air  Force  Systems  Acquisition 
Review  Council  (AFSARC)  T&E  Briefings. 
Lead  Pre-AFSARC  T&E  briefing  preparations 
after  certification  of  readiness  for  dedicated 
lOT&E  (See  paragraph  4.6.2) 

3.9.  Milestone  in  -  Production  Approval.  Senior 
leadership  and  the  Designated  Acquisition  Authority 
evaluate  the  program  and  decide  if  the  program  should 
enter  full-rate  production  and  approval  into  Phase  III, 
Production  and  Deployment 

3.10.  Phase  III  -  Production  and  Deployment.  Each 
MAJCOM  is  responsible  for  planning  and  conducting 
FOT&E  over  the  life  of  the  system,  to  ensure  the  system 
maintains  its  effectiveness  and  suitability  and  meets  the 
users  requirements  as  threats,  operational  environment, 
technology,  mission,  or  employment  techniques  change 
(see  Paragraphs  4. 1-4.6  for  Test  Plan  Procedures). 

•  AFOTEC  may  be  tasked  to  plan/conduct/report 
on  FOT&E  to  answer  specific  issues  directed 
by  the  Milestone  Decision  Authority  after  the 
MS  III  decision 

•  In  the  absence  of  AFOTEC  conducted 
FOT&E,  the  MAJCOM  assumes  primary 
responsible. 


3.10.1.  MAJCOMswill: 

•  Identify  shortfalls  or  deficiencies  to  the  system 
which  impact  mission  effectiveness  or 
suitability,  in  effect,  possibly  documenting  a 
new  mission  need  or  need  for  major 
modification  (see  paragraph  4.4  for  Test 
Reporting  Procedures) 

•  Maintain  the  currency  of  the  OT&E  portions 
of  the  TEMP 

•  FOT&E  results  may  be  the  basis  for  a  MS  IV, 
or  new  MS  0,  decision. 

3.11.  Pre-MS  FV  Major  Modification  Approval,  or 

New  MS  0. 

•  Update  or  regenerate  TEMP 

•  If  FOT&E  results  were  the  basis  for 
identifying  a  deficiency  and/or  creating  a 
mission  need  AF/TE  may  task  AFOTEC  to 
conduct  an  assessment  of  FOT&E  results,  or  a 
separate  OUE 

3.12.  Post  MS  lY. 

•  AFOTEC  evaluates  the  MS  decision  and 
recommends  a  test  strategy  and  test  agency  to 
AF/TE 

•  AF/TE  designates  the  test  agency  (AFOTEC 
vs  MAJCOM)  and  test  strategy  (I/QOT&E  or 
FOT&E)  to  support  future  decisions 


Chapter  4 

OT&E  DOCUMENTATION  AND  BRIEFINGS 


4.1.  Developing  the  Test  Concept 

•  Test  planners,  using  the  strategy-to-task 
framework  outlined  in  Attachment  4, 
determine  the  test’s  purpose,  scope, 
operational  tasks,  and  test  scenarios  as  derived 
from  the  COE  A  MOEs  and  MOPs.  They 
develop  in  outline  form  a  plan  which 
documents  effective  and  efficient  use  of  test 
resources,  facilities,  ranges,  analysis 
techniques,  and  modeling  and  simulation. 
Rough  estimates  are  made  of  test  scenarios. 


threat  laydowns,  schedule,  resource 
requirements,  and  test  limitations.  These 
elements  form  the  test  concept,  a  basis  for  the 
executable  OT&E  plan. 

For  OSD  oversight  programs,  OSD/DOT&E 
requires  a  test  concept  briefing  (at  least  120 
days  prior  to  test  start).  HQ  USAF/TE  must 
preview  and  approve  all  test  concept  briefings 
prior  to  OSD/DOT&E  presentation. 


4.2.  Developing  the  OT&E  Plan. 
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•  The  OT&E  plan  translates  the  test  concept  into 
testable  events/scenarios,  resources,  and 
responsibilities 

•  The  OT&E  plan  describes  everything  needed 
to  complete  an  independent,  operationally 
realistic  evaluation  of  the  weapon  system, 
including  discrete  procedures  for  collecting 
data  and  specialized  analysis  techniques  for 
evaluating  data.  The  outline  of  a  test  plan  is 
shown  at  Attachment  2, 

4.3.  Approving  and  Publishing  the  OT&E  Plan. 

•  The  test  manager/director  is  responsible  for 
developing  and  publishing  (once  approved), 
the  test  plan.  As  a  minimum,  the  plans  must 
be  approved  by  the  commander  of  the 
operational  test  agency. 

•  Draft  test  concepts  and  draft  test  plans  should 
be  provided  to  the  system  program  office  at 
least  one  year  prior  to  test  start.  Final  test 
plans  are  required  prior  to  PEO/DAC 
certification  of  readiness  for  dedicated 
lOT&E. 

•  For  OSD  oversighted  programs,  OSD/DOT&E 
must  approve  test  plan  adequacy  (required  60 
days  prior  to  test).  HQ  USAF/TE  will  review 
all  Air  Force  test  plans  proceeding  to  OSD  and 
will  approve  their  release. 

•  Final  test  plan  distribution  should  be  made  to  all 
involved  parties  and  will  be  program 
dependent.  In  addition,  the  test 

manager/director  must  submit  two  copies  of  the 
approved  OT&E  plan  to  the  Defense  Technical 
Information  Center  (DTIC) 

4.4.  Test  Director  Reporting  Responsibilities. 
OT&E  reports  are  structured  to  provide  a  test  director  the 
flexibility  to  provide  feedback  to  the  appropriate  level  at 
significant  stages  of  the  test,  immediately  following 
completion,  or  as  a  comprehensive  summary  of  the  entire 
test  effort.  When  accomplishing  test  planning,  the  test 
director  or  manager  must  review  the  test  effort  and 
determine  which  reports  are  appropriate  and  when  they 
should  be  provided.  These  report  requirements  are  then 
documented  in  the  TEMP  and  test  plan. 

4.5.  Types  of  OT&E  Reports.  These  reports  are  exempt 
from  report  control  symbol  (RCS)  licensing  in  accordance 
with  AFI  37-124,  The  Information  Collections  and 
Reports  (ICR)  Management  AProgram;  Controlling 


Internal,  Public,  and  Interagency  Air  Force  Information 
Collections. 

4.5.1.  Status  Report.  Status  reports  provide  periodic 
updates  and  important  test  findings  during  OT&E.  Test 
Directors  submit  status  reports  in  letter  or  message  form 
with  the  contents  adjusted  to  meet  individual  program 
needs.  Status  Reports  can  be  submitted  periodically  or 
when  specific  events  dictate. 

4.5.2.  Significant  Test  Event  Report.  This  Air  Force 
report  applies  to  major  test  events  (i.e.  missile  or  aircraft 
launches)  listed  in  the  TEMP.  The  test  director  must 
consider  political  sensitivity,  public  interest,  etc.  when 
determining  program  test  events  that  require  a  report.  The 
operational  test  agency  must  submit  the  report  to  HQ 
USAF/TE  within  24  hours  of  the  test  event. 

4.5.3.  Annual  Report.  Test  directors  submit  annual 
reports  for  tests  lasting  longer  than  one  year.  Reports  are 
sent  to  the  appropriate  HQ  for  relay  to  HQ  USAF/TE  and 
DOT&E  (if  necessary).  Testers  format  and  coordinate 
annual  reports  the  same  as  final  reports. 

4.5.4.  Interim  Summary  Report.  An  interim  summary 
report  is  provided  when  the  final  report  can  not  be 
completed  within  45  calendar  days  of  a  milestone  or 
significant  program  decision.  This  report  (usually 
message  format)  summarizes  OT&E  results  in  detail  to 
support  the  decision.  The  test  director  should  limit 
interim  summary  report  distribution  to  final  report 
recipients. 

4.5.5.  Final  OT&E  Report.  The  test  director  is 
responsible  for  preparing  the  final  OT&E  report.  This 
report  is  an  executive-level  document.  The  final  report 
answers  COIs  and  reports  operational  effectiveness  and 
suitability.  The  report  includes:  test  results,  conclusions, 
and  recommendations;  and  comments  on  constraints, 
limitations  and  any  operational  impacts  found.  It  also 
includes  prioritized  DRs  and  status  to  be  carried  forward 
to  the  operating  command.The  final  report  must  be 
completed  and  approved  no  later  than  60  calendar  days 
after  the  last  test  event.  The  report  provides  test 
information  to  decision  makers,  planners,  and  operators. 
It  also  provides  a  formal,  permanent  record  of  the  results 
of  all  phases  of  OT&E.  The  current  phase  of  OT&E  will 
be  reported  in  the  final  report,  with  annexes  consisting  of 
executive  summaries  of  any  OAs  or  previous  phases  of 
OT&E.  It  is  written  objectively  and  should  relate  test 
results  to  user  criteria.  Test  reports  should  provide 
detailed  technical  information  as  necessary  in  separate 
data  documents.  Refer  to  Attachment  3  for  final  report 
format  and  content.  Refer  to  paragraph  4.6  for  required 
briefings  associated  with  the  final  test  report. 
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•  OSD  Oversight  Programs.  For  OSD  T&E 
oversight  programs,  send  four  copies  of  the  final 
report  to  HQ  USAF/TE.  HQ  USAF/TE  will 
distribute  final  reports  to  DOT&E  and  Dir,  T&E, 
Test  agencies  will  forward  all  other  reports  to 
HQ  USAF/TE  for  information  purposes  upon 
publication. 

•  Release  Authority  Within  the  DoD.  Test  directors 
have  release  authority  to  offices  within  DoD  for  on¬ 
site  OT&E  program  information  with  concurrence  of 
their  commander.  All  releases  of  technical  data 
including  plans  and  reports  must  be  I  AW  AFT  61- 
204,  Controlling  the  Distribution  of  Classified  and 
Unclassified  Scientific  and  Technical  Information. 
Test  Directors  cannot  release  classified  information 
except  as  specified  in  DoD  5200.1,  Information 
Security  Program  Regulation  ,  and  associated 
documents. 

•  Release  Authority  Outside  the  DoD.  The  test 
director  does  not  have  authority  to  release  OT&E 
information  outside  DoD  channels.  Freedom  of 
Information  Act  (FOIA)  requests  should  be  processed 
according  to  AFI  37-131,  Air  Force  Freedom  of 
Information  Act  Program.  Personnel  involved  in 
testing  activities  must  know  the  releasability 
restrictions  for  all  test  communications, 

•  Test  agencies  manage  releases  to  Congress,  the 
General  Accounting  Office,  the  Office  of 
Management  and  Budget,  or  similar  agencies  lAW: 
AFI  90-401,  Air  Force  Relations  With  Congress.,  and 
AFI  65-401,  Air  Force  Relations  With  the  General 
Accounting  Office. 

•  The  appropriate  Public  Affairs  office  must  approve 
release  of  significant  or  potentially  controversial 
information, 

•  The  Information  Branch  of  the  Office  of  the  Vice 
Chief  of  Staff  (HQ  USAF/CVAII)  will  release 
information  to  NATO  and  foreign  nationals.. 

4.6.  Briefing  Requirements. 

4.6.1.  Air  Force  System  Acquisition  Review  Council 
(AFSARC)  T&E  Briefing.  All  AFSARC  MS  I-IV  and 
designated  program  reviews  include  presentations  by  the 
command  owning  the  requirement,  test  agencies  involved 
in  test,  and  the  Program  Executive  Officer  (PEO/DAC  or 
PM).  The  T&E  briefing  is  a  joint  AFOTEC/RTO 
presentation.  The  RTO  is  lead  for  the  T&E  briefing  up  to 


and  including  Certification  of  Readiness  for  Dedicated 
OT&E.  AFOTEC  assumes  lead  after  certification.  Items 
briefed  will  vary  depending  on  where  the  program  is  in 
the  acquisition  process. 

•  During  Phase  I  -  The  briefing  should  cover  the 
overall  test  and  evaluation  strategies  including  DT, 
CT,  and  OT&E 

•  During  Phase  II  -  The  briefing  should  cover  the 
results  of  testing  to  date. 

•  During  Phase  III  -  The  briefing  focuses  on  the  results 
of  testing  in  terms  of  effectiveness  and  suitability  to 
meet  the  users  need  and  plans  for  deficiency 
resolution. 

•  For  program  reviews  -  Briefing  content  should  be 
adjusted  to  support  the  review 

4.6.2.  Final  Report  Briefing  Requirements.  Test 
Directors  may  not  have  sufficient  time  to  publish  final 
reports,  and  may  use  formal  briefings  to  supplement 
major  milestones  or  decision  points.  Testers  will 
summarize  OT&E  results  in  an  executive-level 
presentation. 

4.6.2. 1,  For  all  test  programs: 

•  The  test  director  will  brief  results  of  the  test  effort  to 
the  developer.  The  developer,  in  turn,  will 
recommend  solutions  or  alternatives  to  problems 
identified  during  the  test. 

•  The  test  director  provides  a  tailored  briefing  to  the 
using  MAJCOM 

4.6.2.2.  For  OSD-Oversight  and  HQ  USAF  directed  test 
programs,  the  Test  Director  and  a  SPO  representative  will 
present  a  combined  briefing  focusing  on  both  the 
problems  and  required  fixes  to  HQ  USAF/TE,  SAF/AQ, 
and  appropriate  Air  Staff  offices.  In  addition,  for  OSD- 
Oversight  programs,  HQ  USAF/TE  will  approve  the 
briefing  for  presentation  to  OSD/DOT&E.  Briefers  will 
structure  their  presentation  to  include: 

•  Significant  results  and  conclusions 

•  Recommendations 

•  Deficiencies  and  corrective  actions 

• 


Planned  future  tests 
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Chapter  5 
RESOURCES 


5.1.  Modeling  and  Simulation.  Maximum  use  of  M&S 
will  be  made  to  assist  in  test  planning  and  execution. 
Where  feasible,  M&S  will  be  used  to  predict  test  results 
and  to  better  understand  test  parameter  sensitivities. 
M&S  will  also  be  used  to  the  maximum  extent  possible  in 
test  execution  to  ensure  an  efficient  OT&E  program. 
Specific  uses  of  M&S  will  be  discussed  as  part  of  the  test 
concept  for  each  major  program.  When  feasible,  the  test 
concept  should  include  appropriately  balanced  modeling, 
simulation,  and  field  testing. 

5.2.  Test  Resources  Planning.  Operational  testers  must 
plan  for  all  the  resources  necessary  to  conduct  OT&E. 
Test  resources  are  program  unique  and  include  test 
articles,  funds,  personnel,  facilities,  support  assets  (i.e. 
aircraft),  and  M&S.  The  operational  testers,  in 
conjunction  with  the  single  manager,  are  responsible, 
through  the  TPWG  process,  for  determining  test  resources 
required  for  each  program.  Single  managers  will  acquire 
or  develop  resources  that  do  not  exist.  Test  organizations 
must  identify  their  resource  requirements  in  sufficient 
time  to  permit  inclusion  in  the  TEMP. 

5.3.  Test  Resource  Acquisition  Process.  The  PEOs  and 
Air  Staff  provide  guidance  and  management  of  test 
resource  acquisition.  Resource  identification, 
programming,  budgeting,  and  appropriation  are  integral 
parts  of  test  planning  and  weapon  system  development. 
Test  Investment  Planning  and  Programming  (TIPP) 
Process,  and  the  Central  Test  and  Evaluation  Investment 
Program  (CTEIP)  facilitate  the  acquisition  of  test 
resources.  Refer  to  API  99-109,  Test  Resource  Planning, 
for  more  information. 

5.4.  Resource  Usage.  Testers  must  plan  and  conduct 
tests  to  take  full  advantage  of  existing  or  programmed  Air 
Force  test  resources.  Testers  should  reuse  test  assets 
when  possible.  Testers  will,  in  the  TEMP,  identify  the 
use  of  DoD  or  other  government  resources  when  suitable 
Air  Force  test  resources  are  unavailable.  Use  of  non 
government  test  resources  must  be  justified  in  the  TEMP. 
Testers  requiring  use  of  NASA  facilities  must  submit  a 
request  through  the  appropriate  Air  Force  liaison  office. 


5.5.  Commonalty  and  Interoperability.  New  T&E 
resources  must  comply  with  standards  and  architectures 
established  for  interoperability  and  commonalty  as  well  as 
existing  reliance  agreements.  The  Test  Director  must 
establish  provisions  for  the  protection  of  design 
technologies  and  operational  capabilities.  Refer  to: 
Department  of  Defense  Directive  (DODD)  4630.5, 
Department  of  Defense  Instruction  (DODI)  4630.8,  and 
Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI) 
6212.01. 

5.6.  Test  Range  Requirements.  Test  planners  will 
identify  all  test  range  requirements  as  early  as  possible. 
Ranges  and  centers  are  subject  to  DoD  uniform  funding 
policy  and  require  documentation  of  test  requirements. 
Submit  a  program  introduction  document  (PID)  to 
provide  advanced  range  scheduling,  and  identify  new 
range  capabilities  required.  Consider  all  potential  test 
sites  before  final  selection.  Test  range  users  document 
range  support  requirements  using  the  Universal  Document 
System  (UDS)  for  ranges  that  access  UDS.  Refer  to  UDS 
Handbook  501-79  for  format  information,  available  from 
WSMR/STEWS-NRP,  White  Sands  Missile  Range,  New 
Mexico  88002. 

5.7.  OT&E  Funding.  OT&E  funding  includes  two  types 
of  appropriations,  3600  and  3400  funds.  The  type  of 
OT&E  determines  which  type  of  appropriation  is  used. 
lOT&E  funding  (3600  money)  is  programmed  and 
budgeted  by  AFOTEC.  QOT&E  and  FOT&E  funding 
(3400  money)  are  programmed  and  budgeted  for  by  the 
command  responsible  for  test  conduct.  Test  planners 
must  provide  timely  requirements  to  the  MAJCOM  T&E 
or  budget  office  to  meet  budget  submission  dates  (usually 
24  to  30  months  before  the  fiscal  year  involved).  Budget 
planners  will  then  include  these  requirements  in  the 
MAJCOM’s  O&M  budget  and  POM  submission.  When 
OT&E  resource  requirements  change  significantly, 
resource  managers  must  take  action  to  assure  timely 
OT&E  budget  support  can  be  provided.  Refer  to:  API 
65-601,  Volume  1,  US  Air  Force  Budget  Policies  and 
Procedures, 


HOWARD  W.  LEAF,  Lt  Gen,  USAF  (Retired) 
Director,  Test  and  Evaluation 
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ADDITIONAL  ACQUISITION  AND  TEST  RELATED  PUBLICATIONS 
Joint  Pub  1-02,  Department  of  Defense  Dictionary  of  Military  and  Associated  Terms ,  1  May  1988 

DoD  Directive  4630.5,  Compatibility,  Interoperability,  and  Integration  of  Command,  Control,  Communications,  and 
Intellignece  (C3I)  Systems,  12  November  1992 

DoD  Instruction  4630.5,  Procedures  for  Compatibility,  Interoperability,  and  Integration  of  Command,  Control, 
Communications,  and  Intelligence  { C3I)  Systems 

DoD  Directive  5000.1,  Defense  Acquisition,  23  February  1991 

DoD  Instruction  5000.2,  Defense  Acquisition  Management  Policies  and  Procedures,  23  February  1991 
DoD  Directive  5000.49,  Defense  Acquisition  Board,  1 1  September  1989 
DoD  STD  2167,  Defense  System  Software  Development 

Title  10  US  Code 

NSACSS  Circular  80-17,  System  Test,  Evaluation  and  Transition 

AFI 10-602,  Determining  Logistics  Supportability  and  Readiness  Requirements 

AFI 10-1011,  Information  Security  Program 

AFI  10-1202,  Space  Test  Program  (STP)  Management 

AFI  14-208,  Intelligence  Support  to  the  Air  Force  Acquisition  Process 

AFI  16-301,  US  Air  Force  Priority  System  for  Resources  Management 

AFI  16-501,  Control  and  Documentation  of  Air  Force  Programs 

AFI  21-101,  A/r  Force  Management  Maintenance 

AFPD  21-2,  General  Policy  for  Nonnuclear  and  Nuclear  Munitions 

AFI  21-201,  Inspection,  Storage,  and  Maintenance  of  Nonnuclear  and  Nuclear  Munitions 

AFI  33-104,  C4  Systems  Base  Level  Planning  and  Implementation 

AFI  33-117,  Visual  Information  (VI)  Management 

AFI  33-208,  Classification  Guide  for  COMSEC  and  TEMPEST  Information 
AFMAN  37-139,  Records  Disposition  —  Standards 
AFI  61-202,  Air  Force  Technical  Publications  Program 
AFI  62-201,  Systems  Survivability 

AFI  63-104,  Aircraft-Stores  Certification  Program  (Seek  Eagle) 

AFI  91-202,  The  US  Air  Force  Mishap  Prevention  Program 
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AFI 91-204,  Investigating  and  Reporting  Mishaps 

AFI 91-205,  Nonnuclear  Munitions  Safety  Board 

AFPD  99-1,  Test  and  Evaluation  Process 

AFI  99-101,  Developmental  Test  and  Evaluation 

AFI  99-103,  The  Air  Force  Test  Process 

AFI  99-105,  Live  Fire  Test 

AFI  99-106,  Joint  Test  and  Evaluation 

AFI  99-108,  Programming  and  Reporting  Missile  and  Targets  in  Test  and  Evaluation 
AFI  99-109,  Test  Resource  Planning 
MIL-STD-882B,  System  Safety  Program  Requirements 

MIL-STD-1472,  Human  Engineering  Requirements  for  Military  Systems,  Equipment,  and  Facilities 
T.O.  00-35D-54,  USAF  Material  Deficiency  Reporting  and  Investigation  System 
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OT&EPLAN  FORMAT 


1.  GUIDELINE-OT&E  Plan  Format 

2.  OT&E  Plan  Contents  (see  figure  A2.L). 


CONTENTS 

SECTION  PAGE 

EXECUTIVE  SUMMARY . 

HGURES . 

TABLES . 

ABBREVIATIONS . 

I  INTRODUCTION . 

1.0  GENERAL . 

1 . 1  SYSTEM  INFORMATION . 

LLl  Background . 

LL2  Description . 

1 .2  OPERATIONAL  ENVIRONMENT . 

1 .2. 1  Threat  Summary . 

1 .2.2  Operational  Concept . 

1.2.3  Maintenance  Concept . 

1.2.4  Training  Concept . 

1 .3  PROGF^AM  STRUCTURE . 

II  OT&E  OUTLINE . 

2.0  CRITICAL  OPERATIONAL  ISSUES  AND  OBJECTIVES  . 

2. 1  SCOPE  AND  TEST  CONCEPT . 

2.2  PLANNING  CONSIDERATIONS  AND  LIMITATIONS  . 

2.2. 1  Planning  Considerations . 

2.2.2  Limitations . . . 

2.2.3  Estimated  Cost .  2.3 

SYSTEM  CONTRACTOR  INVOLVEMENT  . 

2.4  OT&E  SCHEDULE  AND  READINESS  REQUIREMENTS  . 

III  METHODOLOGY 

3.0  GENERAL . 

3.0, 1  COI  Summary . 

3.0.2  COI  and  MOE/MOP  Matrix . 

3.1  COM . 

3.1.1  Scope . 

3.1.2  Measures  of  Effectiveness/Performance  and 

Evaluation  Criteria . 

3.1.3  Mission  Scenarios . 

3.1.4  Method  of  Evaluation . 


Figure  A2.1.  Test  Plan  Contents. 
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CONTENTS  (continued) 


SECTION 


PAGE 


3.2  COI-2 . 

3.x  SURVIVABILITY  ASSESSMENT 


ADMINISTRATION . 

4.0  TEST  MANAGEMENT . 

4.1  TASKING . 

4.2  TRAINING  REQUIREMENTS . . 

4.3  SAFETY . 

4.4  ENVIRONMENTAL  IMPACT . 

4.5  SOURCE  SELECTION  INFORMATION . 

4.6  SECURITY . 


REPORTING . 

5.0  REPORTS . 

5.1  BRIEHNGS . 

5.2  DEHCIENCY  REPORTS. 


OPTIONAL  SUPPLEMENTS 


INTELLIGENCE  AND  THREAT  ASSESSMENT  (CLASSIHED) 

OPERATIONS  SECURITY . 

CLASSIFIED  EVALUATION  CRITERIA . 

DETAILED  SYSTEM  DESCRIPTION . 

NOT  USED . 

NOT  USED . 

SOFTWARE  EVALUATION . 

HUMAN  FACTORS . 

WEATHER . . 

MODELING  &  SIMULATION . . . 


GLOSSARY  OF  TERMS . 

REFERENCES . 

DISTRIBUTION . 


Figure  A2.2.  Test  Plan  Contents  (continued). 


SECTION  I  -  INTRODUCTION 


1.0  GENERAL.  Discuss  why  the  test  is  being  conducted 
(to  support  a  production  decision,  to  support  preplanned 
product  improvement,  etc.),  who  will  conduct  the  test,  and 
when  and  where  it  will  occur.  Also  discuss  how  the 
OT&E  effort  fits  into  the  system’s  acquisition  cycle  and 
procedures  for  revising  the  plan  as  both  system  and  test 
approach  maturity. 


1-1  System  Information. 

1-1.1  Background.  Briefly  introduce  the  acquisition 
program,  reason  for  the  program  (whether  it  is  to  replace 
or  upgrade  a  system),  the  acquisition  category,  and 
previous  OT&E.  Identify  any  important  deficiencies  or 
problem  areas  (including  testing  not  accomplished)  from 
previous  T&E.  Cite  the  authority  for  OT&E,  its  priority 
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and  precedence,  and  any  decision  milestones  that  the 
OT&E  will  support.  List  other  relevant  program 
documents  (e.g.,  MNS,  ORD,  PMD,  TEMP,  COEA, 
ADM).  Provide  relevant  information  drawn  from  OT&E 
and  OT&E-related  activities  to  date  (e.g.,  OAs,  OUEs, 
Operational  Feasibility  Demonstrations,  and  other  unique, 
program-specific  activities). 

1.1.2  Description.  In  this  section,  trace  development  of 
the  new  system  to  the  current  model  and  identify:  the 
proposed  system,  how  it  works,  its  major  interfaces, 
whether  it  must  operate  in  a  high-threat  environment,  how 
many  units  the  Air  Force  might  need,  which  command 
and  type  of  personnel  will  operate  it,  any  known  unique 
maintainability  issues,  and  most  importantly,  the  primary 
mission.  Provide  enough  information  to  enable  personnel 
unfamiliar  with  the  system  to  understand  the  COIs. 

•  Briefly  describe  the  system  to  be  tested. 
Consider  the  use  of  photographs  and  diagrams. 

•  If  appropriate,  describe  how  the  system  or 
subsystem  interoperates  with  and  relates  to  other  systems. 

•  If  applicable,  identify  different  types  of 
systems  in  which  a  subsystem  will  be  installed  and 
identify  interfaces  with  the  outside  world,  stating  the 
source  of  its  wartime  tasking.  Include  the  names  of  the 
contractors  doing  concept  exploration  and  designs,  major 
trade  studies  that  are  being  conducted  on  the  program. 
Include  the  SPO's  assessment  of  the  program  technical 
risk,  the  management  strategy  for  system  development, 
and  photographs,  drawings,  or  sketches  of  the  primary 
system  components.  Also  address  program  competition 
and  identify  source  selection  sensitive  (SSS)  information. 

•  Describe  flight  profiles  (if  applicable), 
operations,  and  existing  procedures  being  modified.  List 
significant  characteristics  of  the  test  item  (performance, 
size,  capability,  etc.).  Identify  known  differences 
between  the  test  item  and  anticipated  production 
hardware. 

•  Identify  locations  where  testing  will  be 
conducted  and  planned  test  deployments.  Provide 
rationale  for  particular  test  area  or  range  selection. 

1.2  OPERATIONAL  ENVIRONMENT.  Briefly 
summarize  threat,  operations  concept,  maintenance 
concept,  and  general  training  requirements  developed  by 
the  operating  command.  Summarize  software  support 
concept  or  plan  (when  applicable)  if  not  included  in  the 
maintenance  concept. 

1.2.1  Threat  Summary.  Highlight  the  current  and 
postulated  threats  the  system  will  encounter  from  IOC 


through  IOC  plus  10  years  and  against  which  the  system 
should  be  tested.  This  is  an  integral  part  of  the  OT&E 
plan.  It  may  consist  of  a  short,  unclassified  summary 
paragraph  in  the  body  of  the  plan  and  an  additional 
classified  supplement.  When  appropriate,  the  executive 
summary  of  the  STAR  may  be  used.  In  rare  cases,  only 
the  unclassified  paragraph  will  be  necessary.  If  a 
supplement  is  used,  it  will  generally  contain  several 
sections:  a  listing  of  significant  system  particulars,  a  brief 
system-specific  threat  summary,  threat-related 
documentation  deficiencies,  a  listing  of  the  most 
significant  threats,  coverage  of  the  projected  threat 
environment,  and  a  bibliography  of  the  available  pertinent 
threat  documents. 

L2.2  Operational  Concept: 

•  Describe  the  primary  mission  scenarios  and 
wartime  use  of  the  system  as  stated  in  the  ORD,  COEA, 
MNS,  PMD,  TEMP,  and  messages.  This  may  include 
primary  and  alternate  missions,  system  interfaces,  day  or 
night  sortie  rates,  surge  rates,  accuracy,  system  and 
weapons  numbers,  altitudes,  profiles,  duration,  hours  per 
day,  numbers  of  operators,  dormancy,  shelf  life, 
durability,  typical  environment,  etc. 

•  Possibly  include  in  this  section  one  or  more 
drawings  to  graphically  portray  to  the  reader  how  the 
system  will  be  used  in  an  operational  environment. 

1.2.3  Maintenance  Concept: 

•  Briefly  describe  the  user's  maintenance 
philosophy  for  peacetime  and  wartime  environments  (if 
different)  as  stated  in  the  ORD. 

•  As  a  subset  of  the  maintenance  concept,  briefly 
describe  the  concept  for  depot-level  system  support. 

1.2.4  Training  Concept.  State  who  will  define  and 
develop  training  given  to  personnel  operating  and 
maintaining  the  system  when  it  becomes  operational.  List 
the  numbers  of  O&M  personnel,  skill  levels,  and  training 
required  by  HQ  Air  Education  and  Training  Command 
(AETC)  and  the  operating  command.  Address  training 
requirements  not  presently  available  in  the  US  Air  Force, 
training  media/equipment  required,  facilities,  simulators, 
and  resources  needed  to  fill  those  requirements.  Identify 
location,  duration,  and  source  of  all  AETC,  MAJCOM, 
host  base,  and  unit-administered  training.  State  the 
outcome  (skill  level  achieved,  number  of  graduates  per 
month)  of  each  course. 

1.3  PROGRAM  STRUCTURE: 
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•  Briefly  describe  or  show  the  developer's 
acquisition  strategy  from  concept  exploration  through 
production  and  deployment.  Discuss  how  program 
integration  will  impact  OT&E.  If  applicable,  address 
when  source  selection  will  occur  and  whether  it  will 
restrict  information  release  and  program  meetings. 
Identify  program  risks  and  how  they  are  being  mitigated. 
Highlight  areas  of  concern  where  DT&E  could  affect 
system  certification  and  OT&E.  Discuss  whether  the 
required  contractual  provisions  to  support  OT&E  are  in 
the  statement  of  work  (SOW)  and  the  contractor's  test 
plans.  Describe  the  intermediate  levels  of  performance 
and  capability  that  will  be  used  to  document  the 
development  process  (i.e.,  growth  process).  Include 
pertinent  information  drawn  from  OT&E  and  OT&E- 

SECTION II  - 

2.0  CRITICAL  OPERATIONAL  ISSUES  AND 
OBJECTIVES: 

•  List  COIs.  Explain  why  these  issues  are 
critical  to  this  program.  Explain  how  they  were  derived 
and  relate  them  to  validated  requirements  documents. 
New  COIs  may  surface  at  any  time— even  during  testing. 
Test  planners  should  validate  and  update  issues  during 
test  planning  and  conduct. 

•  List  the  OT&E  objectives  and  each  COI  they 
support.  OT&E  objectives  will  be  used  as  organization 
and  planning  tools.  OT&E  objectives  will  be  statements 
of  areas  to  be  examined,  e.g.,  "rapid  on/off-load,"  or 
"capability  to  engage  and  defeat  enemy  aircraft." 

2.1  SCOPE  AND  TEST  CONCEPT: 


related  activities  to  date  or  planned.  Also  include  relevant 
items  from  the  TEMP  program  summary. 

•  List  known  or  projected  program,  test,  and 
production  events  such  as  OAs,  major  DT&E  or  OT&E 
events  and  start/stop  dates,  meetings  of  decision-making 
bodies,  and  milestones  supported.  Describe  each  OT&E- 
related  activity  and  the  M&S  strategy/plan.  Is  there 
adequate  time  for  testing  and  reporting?  Will  production 
representative  test  articles  be  available  for  dedicated 
lOT&E?  List  any  OT&E-related  activities  planned,  e.g., 
OAs,  operational  utility  evaluations,  operational 
feasibility  demonstrations,  and  other  unique,  program- 
specific  activities.  Provide  a  description  of  those 
activities. 


E  OUTLINE 

•  Provide  a  test  overview  and  how  the  test  is 
structured  to  collect  the  information  needed  to  answer  the 
question  of  system  operational  effectiveness  and 
suitability. 

•  List  pertinent  aspects  of  separate  or  combined 
tests;  field  tests  versus  simulations;  use  of  modeling; 
methods  of  verification,  validation,  and  accreditation  for 
modeling  and  simulation;  and  prioritization  of  test  events 
and  test  planning  assumptions.  If  the  OT&E  is  part  of  a 
combined  or  multiservice  test  program,  discuss  the  Air 
Force  OT&E  effort. 

•  A  test  concept  summary  table  (figure  A2.3.) 
may  be  used  to  group  information  such  as:  the  test  phase, 
existing  test  capabilities,  outcome  expected  for  that  phase 
of  testing,  and  COIs  which  will  be  answered. 
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TEST  PHASE 

TEST  CAPABILITY  TO  BE  USED 

OUTCOME 

DEES 

SIMULATION 

3  SCENARIOS 

THREAT  DENSITY 

ADVANCED  THREATS 

LIMITATION:  ARTIFICIAL  CONFIGURATION 

(SEE  SUP  B  FOR  SPECIFIC  THREATS) 

IDENTIFICATION 
RESPONSE  TIME 
PRIORITIZATION 
MULTIPLE  SYMBOLS 

COM 

NELLIS 

10  SORTIES 

EGLIN 

10  SORTIES 

OPERATIONAL  CONFIGURATION 
CONTROLLED  THREATS 

INSTRUMENTED  EC  RANGE 

OPERATIONAL  PROFILES 

LIMITATIONS:  THREAT  DENSITY 
ADVANCED  THREATS 

(SEE  SUP  B  FOR  SPECIFIC  THREATS) 

IDENTIFICATION 
RESPONSE  TIME 
PRIORITIZATION 
MULTIPLE  SYMBOLS 
DISPLAY  RANGE 
AZIMUTH  ACCURACY 
SYMBOL  SPLITS 
MULTIPLE  SYMBOLS 
RELIABILITY 
MAINTAINABILITY 
COMPATIBILITY/ 
INTEROP 

AIRCREW  INPUT 

COM 

COI-2 

COI-3 

COI-4 

AVAILABILITY 

MODEL 

SENSITIVITY  ANALYSES 

UNLIMITED  RUNS 

AVAILABILITY 

COI-3 

Figure  A2.3.  Test  Concept  Summary  Table. 


d.  Test  summation  matrix  (figure  A2.4.).  This  matrix  provides  the  reader  with  an  overview  of  the  relationships 
between  COIs  and  user  requirements,  events,  scenarios,  and  resources.  It  can  be  used  by  test  managers  or  test  directors  as 
part  of  the  internal  briefings  accomplished  before  test  plan  or  test  execution  approval. 


SCENARIO 

TOD 

ALT 

FORMATION 

UNK/ENEMY 

AIRCRAFT 

SORTIES 

THREAT 

RESOURCES 

INSTRUMENTATION 

COIs 

1 

DAY 

LOW 

1 

34 

2  SA-X2 

3  SA-X9 

IMPACT  GRID 

1,2 

2 

DAY 

HIGH 

2V 

8/4 

8 

GROUND 

JAMMER 

FCC  VAN 

3 

DAY 

MED 

2V 

8/4 

28 

AIRBORNE 

JAMMER 

6F-4 

2A-10 

4F-111 

FCC  VAN 

INSTRUMENT  POD 

1,3 

4 

DAY 

LOW 

4V 

4/8 

19 

2  SA-X2 

3  SA-X9 

INSTRUMENT  POD 

5 

DAY 

HIGH 

2V/CARRIER 

4/8 

6 

ALL 

INSTRUMENT  POD 

O 

6 

NIGHT 

MED 

2V 

34 

2  SA-X2 

3  SA-X9 

IMPACT  GRID 

1,2,4 

7 

NIGHT 

LOW 

2V 

8/4 

8 

AIRBORNE 

JAMMER 

6F-4 

2A-10 

4F-111 

FCC  VAN 

1,2 

8 

NIGHT 

HIGH 

2V 

8/4 

28 

GROUND 

JAMMER 

1,4 

9 

NIGHT 

MED 

4V 

4/8 

19 

2  SA-X2 

3  SA-X9 

INSTRUMENT  POD 

1,3 

10 

NIGHT 

LOW 

2V/CARRIER 

4/8 

6 

ALL 

■ 

Figure  A2.4.  Test  Summation  Matrix. 
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2,2  PLANNING  CONSIDERATIONS  AND 
LIMITATIONS. 

2.2.1  Planning  Considerations.  State  assumptions  or 
factors  that  significantly  affected  test  concept  or  scope. 
Include  areas  that  require  workarounds  but  will  allow  the 
test  to  be  successfully  completed.  Also  include  areas  that 
require  additional  management  attention  to  ensure  a 
successful  test. 

2.2.2  Limitations.  List  factors,  for  which  no  suitable 
workaround  exists,  limiting  the  ability  of  the  test  to 
address  operational  requirements.  State  the  impact  of 
these  limitations  on  the  test  team’s  ability  to  verify 
achievement  of  operational  requirements. 

2.2.3  Estimated  Cost.  Provide  a  cost  estimate  addressing 
the  planning,  executing,  analysis,  and  reporting  of  OT&E. 

2.3  SYSTEM  CONTRACTOR  INVOLVEMENT. 
Describe  how  the  system  contractor  will  not  influence  the 
combined  phase  and  dedicated  phase  of  OT&E  supporting 
a  full-rate  production  decision. 

2.4  OT&E  SCHEDULE  AND  READINESS 

REQUIREMENTS: 


•  Show  planned  testing  versus  testing  completed 
to  date.  Discuss  dates  of  key  test  planning  events, 
briefings,  test  readiness  events,  internal  reviews,  Joint 
Reliability  and  Maintainability  Evaluation  Team 
(JRMET),  and  other  relevant  events. 

•  Highlight  those  events,  actions,  and  items 
which  must  be  in  place  or  accomplished  before  testing 
begins  including:  significant  DT&E  events,  certification 
for  dedicated  OT&E,  training,  support  equipment, 
software  and  technical  data  verifications  and  validations, 
and  facility  completion. 

•  Summarize  major  resource  requirements  for 
agencies  participating  in  the  OT&E.  Identify  major 
required  support  items  such  as:  personnel,  ranges, 
automatic  data  processing,  training,  aircraft,  and 
equipment. 

•  Refer  to  the  current  resource  plan  for  operating 
and  supporting  command  resource  requirements,  to  the 
current  TEMP  for  implementing  command  resource 
requirements,  and  the  SOW  for  contractor  resource 
requirements. 


SECTION  m  -  METHODOLOGY 


3.0  GENERAL. 

3.0.1  COI  Summary.  Present  a  COI-by-COI  summary  of 
the  test.  Briefly  describe  what  test  events  are  planned  and 
their  integration.  Consider  using  flow  charts,  block 
diagrams,  or  matrices  to  show  the  relationships  between 
various  T&E  events  and  major  inputs  and  expected 
outputs  of  each  event. 

3.0.2  COI  and  MOE/MOP  Matrix.  A  matrix  can  be  used 
to  show  the  relationship  between  a  large  number  of  COIs, 
MOEs,  and  MOPs. 

3.1  COM.  State  the  COI. 

3.1.1  Scope.  Describe,  in  general  terms,  testing  required 
(number  of  sorties,  etc.)  to  answer  the  COI.  State  areas  to 
be  examined  (terrain-following  capability,  etc.)  to  gather 
data. 

3.1.2  Measures  of  Effectiveness/Performance  and 
Evaluation  Criteria.  List  the  MOEs/MOPs,  associated 
evaluation  criteria,  and  the  document  from  which  the 
requirement  was  extracted.  More  than  one  MOE/MOP 
may  apply  to  a  COI.  An  MOE  will  have  a  two-digit 
identifier  relating  it  to  the  COI  it  supports.  The  first  digit 


is  the  number  of  the  COI;  the  second  indicates  the  MOE’s 
sequence  in  support  of  the  COI.  An  MOP  will  have  a 
three-digit  identifier.  The  first  digit  is  the  number  of  the 
COI,  the  second  indicates  the  MOE  it  supports,  and  the 
third  indicates  the  MOP's  sequence  in  support  of  the  COI. 

3.1.3  Mission  Scenarios.  Include  descriptions  or 
diagrams  of  representative  mission  scenarios  or  the 
proposed  outlines  of  test  events,  indicating  if  they  are  to 
be  actual  missions,  captive-carry  flights,  missile  shots, 
etc.  Scenarios  listed  here  should  be  those  that  are  critical 
to  determining  mission  capability.  When  the  same 
scenario  is  used  for  more  than  one  COI  or  is  discussed  in 
section  II,  it  may  be  referenced  in  this  section. 

3.1.4  Method  of  Evaluation.  Describe  how  data  will  be 
collected  and  how  results  associated  with  the  MOE  and 
MOPs  will  be  combined  to  produce  an  answer  for  each 
COI.  Present  a  sequence  of  events  that  logically  leads  to 
and  supports  the  rating.  If  the  aggregation  methodology 
in  this  paragraph  is  used  for  other  COIs,  reference  this 
paragraph. 

3.2  001-2.  Continue  as  above. 

3.x  SURVIVABILITY  ASSESSMENT: 
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a.  If  survivability  is  a  major  program  concern  as 
indicated  by  PMD  or  operational  requirement,  address  it 
as  a  COL  If  survivability  is  not  a  major  issue,  but  the 
OT&E  team  plans  to  provide  information  about  system 
performance  and  characteristics  that  affect  survivability, 
address  it  in  a  separate  paragraph  much  like  a  COI,  after 
all  COIs  have  been  addressed.  If  MOPs  are  used  for  the 
assessment,  discuss  them  as  stated  above. 

b.  Identify  the  type  of  information  to  be  reported. 
State  where  data  will  be  collected  in  support  of  the 


survivability  assessment.  Explain  how  the  assessment  is 
an  examination  of  the  impact  of  system  characteristics 
and  performance  on  survivability.  This  assessment  will 
involve  the  use  of  all  available  data  including  specifically 
designed  field  testing,  laboratory  testing,  intelligence 
estimates,  vulnerability  analyses,  and  simulation  and 
modeling  results.  Outline  test  activities  that  will  provide 
the  assessment.  For  the  survivability  assessment,  it  may 
be  possible  and  necessary  to  enlist  support  from  outside 
agencies  in  conducting  an  analysis  that  identifies  the 
primary  factors  applicable  to  a  system’s  survivability. 


SECTION  IV  -  ADMINISTRATION 


4.0  TEST  MANAGEMENT.  List  the  agencies  involved 
in  the  test  and  summarize  their  responsibilities  and 
relationships. 

4.1  TASKING.  Identify  the  implementing,  operating, 
participating,  and  supporting  organizations  that  have  a 
direct  input  to  test  planning  or  conduct.  Identify  the  test 
director  and  key  staff  by  organization  and  office  symbol. 
Include  a  test  team  organizational  chart.  Outline  test  team 
chain  of  command  and  individual  responsibilities  for 
scheduling  test  resources. 

4.2  TRAINING  REQUIREMENTS.  Identify  test  team 
training  requirements  to  include:  operator,  maintenance, 
and  specialized  training  (i.e..  Type  1  training  (AFI  36- 
2201,  Developing  Military  Training  Programs),  analyst, 
data  base,  computer  programming,  contractor-provided 
systems  training,  etc.).  Specify  how  and  by  whom  these 
requirements  will  be  satisfied  to  include  a  schedule  for  the 
planned  training. 

4.3  SAFETY.  Ensure  that  required  statements  contained 
in  AFI  91-202,  The  US  Air  Force  Mishap  Prevention 
Program',  AFI  91-204,  Investigating  and  Reporting 
Mishaps',  Military  Standard  (MIL-STD)  882,  System 
Safety  Program  Requirements',  and/or  other  applicable 
MIL-STDs  or  instructions  are  included. 

4.4  ENVIRONMENTAL  IMPACT.  Normally,  the 
implementor  files  an  environmental  impact  statement  for 
DT&E.  Depending  on  DT&E  location  and  planned 
operational  environment,  use  and  reference  this  statement 
as  required.  Where  DT&E  and  OT&E  locations  are 
different,  include  a  separate  statement  to  describe  the 
differences  and  impact  caused  by  OT&E  (reference 
AFPD  32-70,  Environmental  Planning  ).  Contact  the 
local  environmental  coordinator  for  the  test  location  when 
a  separate  statement  is  needed  or  when  the  environmental 
impact  statement  for  the  test  program  is  inadequate.  The 
environmental  coordinator  is  a  part  of  the  civil 
engineering  community  and  can  provide  the  expertise  to 
determine  the  impact  of  the  program,  if  any,  on  the 


environment.  Operational  test  agencies  are  required  to 
comply  with  all  federal,  state,  and  local  environmental 
statutes  when  conducting  OT&E.  The  OT&E  plan  must 
contain  one  of  these  statements: 

•"The  OT&E  will  be  conducted  in  conjunction  with 
(program  name),  and  environmental  impact  is 
covered  by  (implementing  agency)  statement 
(identify  statement)." 

•"All  pertinent  environmental  factors  have  been 
considered,  and  this  headquarters  has  determined  that 
the  planned  OT&E  will  present  minimal  hazards  to 
the  environment." 

•"This  (state  either  headquarters,  command,  or 
agency)  has  determined  that  conducting  this  OT&E 
will  have  no  adverse  effects  on  the  environment." 

4.5  SOURCE  SELECTION  INFORMATION.  Refer  to 
AH  37-131,  Air  Force  Freedom  of  Information  Act 
Program.  List  any  deviations  or  exceptions. 

4.6  SECURITY. 

4.6.1  Information  Protection. 

4.6. 1.1  Classified  Information.  Identify  by  name  and 
date  all  security  classification  guides  governing 
classification  of  information  relating  to  the  system  and 
test  program.  Identify  any  security  master  or  program 
protection  plans  having  applicability  to  the  system  or  test 
program.  Also  state  that  classified  information  will  be 
safeguarded  and  protected  according  to  the  provisions  of 
DOD  5200. 1-R,  Information  Security  Program 
Regulation,  and  its  supplements. 

4.6. 1.2  Unclassified  Information.  Identify  applicable 
protection  measures  for  unclassified  information.  System 
acquisition  program  protection  plans  will  identify 
"Critical  Information"  requiring  protection.  Supplement 
B,  Operations  Security  (OPSEC)  Plan,  to  this  OT&E  plan. 
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also  identifies  the  Critical  Information  protection 
requirements. 

4.6.2  Communications  Security  (CQMSEQ.  When  there 
is  a  need  to  have  a  supporting  COMSEC  Account 
(controlled  cryptographic  items)  identify  the  supporting 
COMSEC  manager  in  this  section.  COMSEC  monitoring 
applies  to  all  DOD  telephones  and  similar 
communications  devices  that,  if  considered  appropriate, 
may  be  identified  in  this  section. 

4.6.3  Control  of  Compromising  Emanations 
(TEMPEST).  Identify  equipment  requirements  for 
processing  classified  information  by  area  or  location. 
Identification  of  any  TEMPEST  countermeasure  applied 
to  the  specific  equipment  may  cause  the  information  of 
this  section  to  be  classified  in  its  own  right,  and  that 
should  be  avoided  when  possible. 

4.6.4  Physical  Security.  When  applicable,  identify  any 
special  physical  security  requirements  applying  to  the  test. 
Refer  to  AFI  31-209,  Air  Force  Resource  Protection 
Program^  and  AFI  31-101,  Air  Force  Physical  Security 
Program. 


SECTION  V- 

5.0.  REPORTS.  This  section  outlines  the  report 
requirements.  Specifically  identify:  timing  of  reports, 
writing  responsibilities,  addressees,  format,  content,  and 
transmission  means.  Give  special  consideration  to 
distribution  of  OT&E  reports  ensuring  OT&E-related 
information  is  released  to  agencies  with  a  valid  need-to- 
know. 

-  STATUS  REPORT 

-  SIGNinCANT  TEST  EVENT  REPORT 

-  ANNUAL  REPORT 

-  INTERIM  SUMMARY  REPORT 

-  FINAL  OT&E  REPORT 


5.1  BRIEFINGS.  Identify  briefings  required  before, 
during,  and  after  OT&E.  Include  a  short  narrative  on  the 
content  and  intended  audience  of  all  briefings. 


4.6.5  Operations  Security  (OPSEC).  Most  acquisition 
programs  have  OPSEC  requirements  applied  by  a 
separate  directive.  Security  Master  or  Program  Protection 
Plan,  and  that  directive  or  plan  should  be  identified  in  this 
section.  Also,  use  one  of  the  following  statements  in  the 
OT&E  plan. 

•"This  program  is  susceptible  to  foreign  intelligence 
exploitation  wherein  classified  or  unclassified  critical 
information  (also  known  as  Essential  Elements  of 
Friendly  Information  (EEFI))  could  be  compromised. 
Refer  to  Supplement  B  for  application  of  OPSEC  to 
this  OT&E  test  program." 

•"This  program  has  been  determined  to  be 
unsusceptible  to  foreign  intelligence  exploitation." 

4.6.6  Security  Awareness.  Identify  any  special 
awareness  or  educational  requirements  desired  by  the  test 
director  or  test  manager  or  mandated  in  related  security 
directives.  Also  insert  "Security  indoctrination  and 
recurring  training  will  be  consistent  with  the  provisions  of 
DOD  5200. 1-R  and  its  implementing  supplements.  When 
applicable,  OPSEC  training,  to  include  a  discussion  of 
critical  information  and  protective  measures,  will  be  part 
of  any  initial  or  annual  security  indoctrination  training." 

REPORTING 

5.2  DEHCIENCY  REPORTS  (DR).  Describe  how 
deficiencies  and  proposed  enhancements  discovered 
during  OT&E  will  be  validated,  reported  and  managed 
per  TO  00-35D-54,  USAF  Material  Deficiency  Reporting 
and  Investigation  System.  At  a  minimum,  the  description 
should  address  the  duties  and  responsibilities  of  the  DR 
monitor,  including  attendance  at  MIP  boards,  the  overall 
management  of  the  DR  process,  and  the  procedures  for 
prioritizing  DRs. 

3.  OT&E  Plan  Supplements.  Supplements  should 
contain  detailed  information  supporting  the  body  of  the 
plan.  The  number  and  type  of  supplements  used  are 
program  dependent.  Supplements  can  also  be  used  to 
"extract"  classified  material  from  the  test  plan  body  to 
simplify  handling  procedures.  The  format  will  vary 
depending  on  program  specifics  such  as  the  number  of 
COIs,  MOEs/MOPs,  etc.  Some  commonly  used 
supplements  and  their  outlines  follow. 
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SUPPLEMENT  A  -  INTELLIGENCE  AND  THREAT  ASSESSMENT 

A.O  GENERAL.  A,4  THREAT-RELATED  COIs.  List  those  COIs  that 

are  threat-driven  here, 

A.1  OPERATIONAL  ENVIRONMENT.  Describe  how 

the  system’s  environment  (altitude,  penetration  of  enemy  A.5  CRITICAL  INTELLIGENCE  PARAMETERS 

airspace,  etc.)  subjects  it  to  various  threats,  (CIP).  Not  all  threat  assessments  will  have  this  section. 

CIPs  are  identified  by  the  intelligence  community  and 
A.2  ENEMY  DOCTRINE.  Describe  how  the  enemy  listed  in  the  System  Threat  Assessment  Report  (STAR), 

would  view  the  importance  of  the  system.  Describe  They  are  on  a  watch  list  of  key  systems  or  tactics  that 

doctrine  and  general  strategies  likely  to  be  employed  to  would  have  significant  bearing  on  the  threat  assessment, 

defeat  the  system.  If  the  intelligence  estimate  regarding  a  CIP  changes,  it 

could  impact  the  overall  threat  assessment  for  the  blue 

A. 3  THREATS.  List  enemy  threat  systems  in  categories  system  being  developed. 

(e.g.,  air-to-air,  surface-to-air,  directed  energy,  etc.). 

A.6  BIBLIOGRAPHY.  List  sources  used  in  the 
supplement  as  well  as  additional  reference  material. 

SUPPLEMENT  B  -  OPERATIONS  SECURITY  (OPSEC) 

B. l  PURPOSE.  B.4  OPSEC  VULNERABILITY  ANALYSIS.  Identify 

all  possible  OPSEC  vulnerabilities,  which  is  any 
B.2  CRITICAL  INFORMATION  or  ESSENTIAL  condition  where  friendly  actions  produce  indicators  that 

ELEMENTS  OF  FRIENDLY  INFORMATION.  may  be  collected  and  evaluated  in  the  intelligence 

collection  process. 

B.3  ASSESSMENT  OF  THE  INTELLIGENCE 

THREAT.  Identify  specific  intelligence  collection  B.5  OPSEC  PROTECTIVE  MEASURES.  Identify 

threats.  Signals  intelligence  (SIGINT),  imagery  OPSEC  training/awareness  requirements  and  insert  each 

intelligence  (IMINT),  human  intelligence  (HUMINT),  or  protective  measure  along  with  its  implementing  guidance, 

an  all-source  multi-discipline  collection  effort  may  form 

the  intelligence  threat  for  any  OT&E  plan.  Test  managers  B,6  OPSEC  LESSONS  LEARNED.  The  test  team  will 

should  seek  the  assistance  of  OPSEC  program  managers,  document  OPSEC  lessons  learned  during  the  test  and 

security  planners,  AFOSI  representatives,  and  intelligence  immediately  forward  them  to  the  OPSEC  officer  for 

program  managers,  in  developing  this  section.  appropriate  staffing.  They  are  included  in  the  final  OT&E 

report. 

SUPPLEMENT  C  -  CLASSIFIED  EVALUATION  CRITERIA 

This  supplement  should  contain  classified  evaluation  criteria  to  be  used  during  OT&E. 

SUPPLEMENT  D  -  DETAILED  SYSTEM  DESCRIPTION 

This  section  contains  additional  system  description  information  necessary  to  supplement  the  OT&E  plan. 

SUPPLEMENT  G  -  SOFTWARE  EVALUATION 

A  separate  software  supplement  may  be  required  to 
provide  specific  guidance  to  the  test  team  deputy  for 
software  evaluation. 

G.l  GENERAL.  G.1.3  Limitations  of  the  Software  Evaluation. 


26 


API  99-102  Attachment  2  22  July  1994 


G.  1.4.2  Responsibilities. 

G.1.5  Software  Support  Concept. 

G.2  SOFTWARE  SUPPORT  TO  SYSTEM 
EVALUATION.  This  portion  of  the  software  evaluation 
supplement  lists  those  COIs  and/or  MOEs/MOPs 
supported  by  the  software  evaluation. 

G.2.1  Method.  Describe  how  software  evaluators  will 
participate  during  the  test. 

G.2.2  Data  Management. 


G.2.2.1  Data  Requirements.  Briefly  describe  the  source 
for  every  data  item. 

G.2.2,2  Data  Collection  and  Processing.  Summarize  the 
format,  labeling,  routing,  and  filing  to  be  done  for  each 
data  source. 

G.2.2.3  Data  Analysis.  Discuss  data  analysis  techniques 
to  be  used  by  software  evaluators. 

G.2.3  Evaluation.  Discuss  the  evaluation  techniques  or 
procedures  to  be  used.  Explain  how  the  software 
evaluation  results  are  to  be  combined  with  other  OT&E 
results  to  determine  system  performance. 

G.3  ADDITIONAL  SOFTWARE  EVALUATION. 


SUPPLEMENT  J  -  MODELING  AND  SIMULATION 


J.O.  PURPOSE.  Brief  statement  of  why  supplement  is 
required. 

J.  1  OVERVIEW.  Summary  narrative  of  overall 
supplement. 

J.2  PROGRAM  M&S  STRATEGY.  Brief  synopsis  of 
weapon  system  acquisition  strategy  with  emphasis  on  the 
role  of  M&S. 

J.3  M&S  APPROACH.  Most  important  part  of 
supplement.  Includes  description  and  supporting 
rationale  for  M&S  use  as  applied  to  requirements 
defmition/analysis,  research  and  development,  test  and 
evaluation,  training  and  operations.  Describe  how  M&S 
has  been  and/or  will  be  employed  throughout  the  program 
by  government  and/or  contractor  activities;  Describe 


verification,  validation,  and  accreditation  (VV&A) 
methods  and  processes. 

J.4  RELATED  M&S  ACTIVITIES.  Describe  supporting 
and/or  related  M&S  efforts  and  how  they  will  be  used  by 
this  program.  Include  supporting  data  requirements  for 
M&S. 

J.5  MANAGEMENT.  Include  wiring  diagrams.  POCs 
for  the  M&S  activity  of  the  program. 

J.6  FACILITIES/EQUIPMENT  REQUIREMENTS. 
Include  description  of  government/contractor  hardware, 
software  and  facility  requirements. 

J.7  SUPPLEMENTAL  INFORMATION.  Include 
miscellaneous  related  information.  Test  Manager 
comments. 
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OT&E  REPORT  FORMAT 


1.  Guidelines  for  OT&E  Report  Format. 

2.  OT&E  Report  Format  and  Contents.  This  section 
outlines  the  preferred  format  and  content  for  a  final 
OT&E  report. 

2.1.  Final  Report  Cover.  The  cover  must  contain  the 
information  shown  in  figure  A3.1.  Use  red  cover  for 
classified  reports.  Mark  all  classified  test  documents 
according  to  DoD  5200. l-R;  AFPD  31-4,  Information 
Security;  and  AEI  31-401,  Information  Security  Program 
Management.  When  both  the  title  and  the  entire  report 
are  unclassified,  markings  are  not  required.  If  the 
document  contains  restricted  information  (for  example, 
restricted  data,  not  releasable  to  foreign  nationals),  mark 
the  cover  accordingly.  Since  colored  pages  will  not 
microfilm  legibly,  leave  the  reverse  side  of  the  cover 
blank. 

2.2.  Title  and  Signature  Page.  A  sample  is  presented  in 
figure  A3.2.  The  title  page  must  repeat  all  the 
identification  and  markings  found  on  the  front  cover.  Do 
not  print  on  the  reverse  side  of  a  colored  cover. 

2.3.  Report  Documentation  Page.  Attach  SF  298,  Report 
Documentation  Form,  to  reports  submitted  to  the  Defense 
Technical  Information  Center  (DTIC).  The  entries  in 
blocks  13  (abstract)  and  14  (subject  terms)  are  important 
for  proper  control  and  release  of  the  document.  For 
example,  if  the  report  concerns  the  OT&E  of  a  jet  fighter 
engine  with  thrust  reversal  capability,  logical  Subject 
Terms  might  include:  "jet  engine,"  "fighter  aircraft 
engines,"  "thrust  reversal,"  "exhaust  nozzles,"  "thrust 
vectoring,"  or  "short  takeoff  and  landing  (STOL)."  The 
Abstract  might  mention:  the  type  of  test  done  (lOT&E, 
etc.),  when  and  where  the  test  was  conducted,  who  the 
contractors  were,  or  any  piece  of  information  that  might 
be  valuable  to  other  DTIC  users.  Information  in  the 
report  must  be  safeguarded,  especially  if  it  is  of  a 
proprietary  nature  or  not  universally  releasable.  Report 


authors  must  carefully  consider  how  information  will  be 
disseminated,  even  after  the  report  has  been  distributed. 
Use  the  distribution  limitation  statements  in  AFI  61-204, 
Controlling  the  Distribution  of  Classified  and 
Unclassified  Scientific  and  Technical  Information,  in 
addition  to  the  report  documentation  page  entries  to 
ensure  the  report  is  available  only  to  people  and  offices 
with  a  valid  need-to-know.  To  allow  legible 
microfilming,  do  not  reproduce  this  page  on  colored 
paper. 

2.4.  Report  Contents.  The  report  contents  is  shown  in 
figure  A3.3. 

2.4.1.  Figures  and  Tables.  Provide  separate  lists  by  page 
number.  Number  all  figures  and  tables  consecutively 
throughout  the  sections.  Use  short  and  descriptive  titles. 

2.4.2.  Abbreviations.  Explain  abbreviations  and  symbols 
so  they  are  readily  understood  by  the  reader.  The  heading 
for  this  page  will  include  only  the  applicable  items;  for 
example,  if  no  symbols  are  included,  the  heading  would 
be  "ABBREVIATIONS". 

2.4.3.  Glossary.  If  necessary,  provide  a  glossary  of 
terms  that  may  be  unfamiliar  to  the  reader.  Common  Air 
Force  terms  need  not  be  defined. 

2.4.4.  References.  If  necessary,  provide  references. 

2.5.  Distribution.  Send  two  copies  of  all  final  OT&E 
reports  to  DTIC.  Other  distribution  requirements  should 
include  those  agencies  participating  in  the  test  and  the 
history  office  of  the  conducting  organization’s 
headquarters. 

2.6.  Report  Format.  The  remainder  of  this  attachment 
illustrates  the  required  format  for  a  final  OT&E  report. 
Discuss  COIs  in  the  same  order  as  presented  in  the  plan. 
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EXECUTIVE  SUMMARY 


The  executive  summary  should  summarize  (ideally  two 
pages,  single  spaced)  the  report  and  explain  the  OT&E 
and  significant  results.  The  executive  summary  should: 
briefly  state  the  purpose  of  the  OT&E,  describe  the  item 


tested  and  its  purpose  or  mission,  provide  system 
background,  summarize  results  by  COI,  summarize  the 
status  of  service  reports,  and  provide  concluding  remarks. 


SECTION  I  -  PURPOSE  AND  BACKGROUND 


1.0  OT&E  PURPOSE.  Provide  a  concise  statement  of 
OT&E  purpose. 

1.1  AUTHORIZING  DIRECTIVES.  Cite  the  authority 
for  the  OT&E,  its  priority,  and  precedence.  Identify 
program  documentation  such  as  the  MNS,  PMD,  etc. 

1.2  OT&E  BACKGROUND.  Provide  a  summary  of 
previous  OT&E  of  the  system  and  important  deficiencies 
or  problem  areas  (including  COIs  not  resolved)  from 
those  OT&Es.  Identify  the  lack  of  previous  test 
information. 


1.3  DESCRIPTION  OF  SYSTEM  TESTED.  Describe 
the  system  tested.  If  differences  exist,  compare  and 
contrast  the  system  tested  to  the  expected  operational 
system. 

1.4  TEST  TEAM,  LOCATION,  AND  DATES.  Identify 
test  team  composition  (including  names  of  key  personnel 
and  average  number  of  personnel  assigned  and  attached) 
and  operating  locations  (including  ranges  and  test 
facilities).  Use  organization  charts  if  available. 

1.5  CLASSIFICATION  STATEMENT.  Identify 
classified  material  and  declassification  procedures  (if 
applicable). 


SECTION  n  -  OT&E  DESCRIPTION 


This  section  contains  a  general  overview  of  how  the 
OT&E  was  conducted. 

2.0  CRITICAL  OPERATIONAL  ISSUES.  List  critical 
operational  issues.  This  list  contains,  in  a  single  location, 
information  reported  throughout  Section  III.  Indicate 
which  COIs  were  resolved  and  were  not  resolved.  The 
information  is  presented  as  a  listing  of  each  COI  and 
MOE/MOP  followed  by  a  correlation  matrix, 

2.1  SCOPE  AND  METHOD  OF  ACCOMPLISHMENT. 
Provide  an  overview  of  how  the  OT&E  was  conducted. 
Write  in  the  past  tense. 


2.2  PLANNING  CONSIDERATIONS  AND  LIMITING 
FACTORS.  Identify  planning  considerations  that  shaped 
the  test  design  and  factors  limiting  realism  or  affecting 
results.  Discuss  any  testing  that  was  not  accomplished 
and  any  follow-on  testing  requirements. 

2.3  CONTRACTOR  INVOLVEMENT.  For  programs 
affected  by  Public  Law  restrictions  on  system  contractor 
participation  during  the  dedicated  phase  of  OT&E, 
explain  how  involvement  by  system  contractor  personnel 
was  controlled. 


SECTION  m  -  OPERATIONAL  EFFECTIVENESS  AND  OPERATIONAL  SUITABILITY 


This  section  is  the  essence  of  the  final  report.  It  details 
the  evaluation  of  the  system  tested,  conclusions  supported 
by  specific  results,  and  recommendations.  The  format  for 
this  section  follows:  paragraph  numbering  for  Section  Ell 
is  designed  so  that  paragraph  3.1  correlates  to  COI-1  and 
paragraph  3.2  correlates  to  COI-2. 

3.0  SUMMARY.  List  the  most  important  conclusions. 
When  rating  OT&E  MOEs  and  MOPs,  "met  user  criteria" 
describes  performance  that  met  or  exceeded  a  stated 
mature  requirement  or  interim  criterion.  "Deficient" 
describes  performance  that  did  not  consistently  meet  a 
mature  requirement  or  interim  criterion.  If  performance 


in  a  particular  area  was  not  tested,  the  results  must  be 
labeled  "not  tested."  This  section  must  focus  the  reader’s 
attention  on  the  most  important  events  that  influenced  the 
ratings.  Present  all  statistical  data  such  as:  missions, 
sorties  and/or  flying  hours  by  participating  aircraft, 
launches,  and  captive-carry  time. 

3.1  COI-1 .  State  COIs,  with  answers,  followed  by  the 
MOEs  and  MOPs  and  the  rating  associated  with  the  each. 
MOEs  may  not  have  ratings  depending  on  whether  they 
had  criteria. 
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3.1.1  Method.  State  the  approach  used  to  answer  this 
COL  Include  statistical  data  that  enumerates:  number  of 
aircraft  or  other  major  equipment  used,  number  of  flying 
hours,  communications  link  time,  etc.  Address  different 
test  locations  separately.  Separate  the  statistics  that  apply 
to  combined  DT&E/OT&E  and  to  dedicated  OT&E. 
When  the  same  method  used  to  test  applies  to  a  number  of 
COIs,  reference  the  paragraph  first  identifying  the 
method. 

3.1.2  Results  and  Conclusions.  State  facts  and 
conclusions  that  logically  follow,  and  can  be  supported 
by,  specific  test  results.  Highlight  significant  capabilities, 
advantages,  and  shortcomings  of  the  system.  Testers  will 
list  results  associated  with  MOEs  and  MOPs  in  order  of 
importance,  not  as  they  are  listed  in  the  OT&E  plan. 

3.1.3  Recommendations.  State  recommendations  in  the 
same  order  as  discussed  in  the  Results  and  Conclusions 


paragraph.  Ensure  recommendations  are  logically  based 
on  the  results.  If  testers  have  no  recommendations,  it 
must  be  noted.  State  the  individual  or  agency  responsible 
for  action  (HQ  AFMC,  other,  MAJCOM,  etc.).  If  the 
recommendation  is  associated  with  a  major  DR,  also 
place  the  DR  number  in  parentheses  at  the  end  of  the 
recommendation  paragraph  (i.e.  DR  II-83001-ALOCTF). 

3.2  COI  2.  Document  additional  COIs  in  the  same 
manner  as  stated  in  paragraph  3.1. 

3.x  SURVIVABILITY  ASSESSMENT.  List  test  results 
and  system  characteristics  that  were  found  to  have  a 
bearing  on  system  survivability. 

3.Y  ENVIRONMENTAL  IMPACTS.  List  any  results 
from  test  or  analysis  that  would  help  the  user  develop  an 
Environmental  Impact  Statement  for  system  deployment 
and  utilization. 


SECTION  IV  -  DEFICIENCY  REPORTS 


A  major  purpose  of  OT&E  is  to  identify  operational 
deficiencies  and  enhancements.  This  section  of  the  final 
report  should  summarize  the  status  of  all  open,  awaiting 
fix  verification,  and  closed  DRs.  The  status  for  closed 
DRs  will  include  who  closed  them  and  how  they  were 
verified.  Include  a  prioritized  list  of  DRs  and  summarize 
any  proposed  enhancements  identified  during  OT&E. 

4.0  DEnCIENCY  REPORT  STATUS.  Summarize  DRs 
in  accordance  with  the  categories  in  TO  00-35D-54. 

a.  Open.  Program  office  actions  have  not  started  or 
are  in  progress. 

b.  Awaiting  Fix  Verification.  Program  office 
actions  are  complete,  and  the  only  remaining  step  is  fix 
verification. 

c.  Closed.  Chose  not  to  fix,  or  fix  is  completed  and 
verified. 

d.  Category  I  (Cat  I).  A  deficiency  that  requires 
immediate  corrective  action  because: 


(1)  The  condition  may  cause  death,  severe 
injury,  severe  occupational  illness,  or  major  system 
damage  or  loss. 

(2)  The  condition  causes  unacceptable  delays 
in  testing  or  prevents  successful  mission  accomplishment 
(because  of  severity  and  frequency  of  the  deficiency)  and 
would  critically  impact  the  combat  readiness  capability  of 
an  operational  unit. 

e.  Category  II  (Cat  II).  A  deficiency  condition  that 
prevents  successful  mission  accomplishment  (system  does 
not  meet  minimum  operational  requirements,  but  does  not 
justify  immediate  corrective  action  in  accordance  with 
Cat  I)  or  degrades  a  system’s  operational  effectiveness 
and/or  operational  suitability. 

4.1  IMPACT  SUMMARY,  Address  the  effect  of 
identified  DRs  on  the  system  and  the  test  program. 

4.2  DEFICIENCY  REPORTS.  Include  a  narrative 
statement  addressing  the  method  for  prioritizing  DRs. 
The  test  director  will  determine  the  number  of  DRs  to  be 
detailed  in  the  report.  When  the  total  list  is  extensive, 
include  only  open  Category  I  DRs  and  those  Category  II 
DRs  that  require  additional  visibility  here.  Include  those 
DRs  not  addressed  in  an  annex  or  data  document. 
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SECTION  V  -  SUMMARY  OF  CONCLUSIONS  AND  RECOMMENDATIONS 

The  format  of  this  section  is  flexible  and  may  be  tailored  to  the  requirements  of  the  particular  report.  Use  a  numbering 
system  similar  to  the  other  sections.  Separate  paragraph  titles  may  not  be  needed  when  the  report  summary  is  provided  in  a 
short  narrative.  Following  are  examples  of  possible  formats  for  the  report  summary: 


Example  1 


Example  2 


Example  3 


5.0.  GENERAL  5.0  GENERAL  5.0  GENERAL 

5.1  EFFECTIVENESS  5.1  CONCLUSIONS  5.1  COM 

5.2  SUITABILITY  5.1.1  Effectiveness  5.1.1  Conclusions 

5.3  CRITICAL  ISSUES  5.1.2  Suitability  5.1.2  Recommendations 

5.2  RECOMMENDATIONS  5.2  COM 

5.2.1  Effectiveness 

5.2.2  Suitability 

5.3  SUMMARY 
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OT&E  PLANNING  WITHIN  THE 
STRATEGY-TO-TASK  FRAMEWORK 


1.  Nature  of  Planning.  Planning  must  be 
comprehensive,  structured,  and  based  on  the  system’s 
intended  mission  and  operational  tasks.  It  must  also 
develop  the  rationale  and  actions  to  be  followed  during 
the  test;  outline  the  procedures  for  collecting, 
documenting,  and  evaluating  the  data  collected;  identify 
all  required  resources;  and  accurately  communicate  the 
proposed  test  program  to  approval  authorities  and  other 
participants.  Successful  test  planning  culminates  in  an 
executable  test  plan  and  a  complete  audit  trail  back  to  the 
Defense  Planning  Guidance. 

2.  Purpose  of  OT&E  Planning.  Proper  OT&E  planning 
facilitates  the  acquisition  of  relevant  data  to  determine 
weapon  system  operational  effective-ness  and  suitability 
in  support  of  system  acquisition  decision  making.  While 
careful  planning  will  not  guarantee  a  successful  test 
program,  inadequate  planning  may  result  in  significant 
test  shortfalls,  cost  and  schedule  overruns,  and  ultimately 
system  failure. 

3.  A  Framework  for  Planning.  The  "strategy-to-task" 
model  shown  in  Figure  A4.1.  is  a  framework  for  test 
planners  to  link  the  system’s  specific  oper-ational  tasks 
and  missions  to  the  national  military  strategy  provided  by 
the  Joint  Chiefs  of  Staff.  Working  from  the  top  down  in 
the  hierarchy,  national  security  strategy  is  supported  by 
national  military  strategy,  which  in  turn  is  supported  by 


various  regional,  or  theater-level,  strategies.  These 
strategies  are  supported  by  regional  operational  objectives 
composed  of  one  or  more  operational  tasks  or  missions. 
Once  the  user  defines  the  operational  tasks  and  missions, 
then  system  performance  parameters,  system 
characteristics,  and  associated  thresholds  and  objectives 
are  determined  and  published  in  the  ORD.  The  user, 
assisted  by  the  tester,  must  also  include  in  the  ORD 
"testable"  measures  of  effectiveness  (MOE)  and  measures 
of  performance  (MOP)  for  objectively  demonstrating 
system  performance  in  operational  test  and  evaluation 
(OT&E).  The  focus  of  operational  testing  is  primarily  on 
the  bottom  two  tiers  of  Figure  A4. 1 .  For  OT&E  planning 
purposes,  the  term  "operational  task"  is  approximately 
equivalent  to  "the  mission." 

4.  Linking  OT&E  to  National  Military  Strategy. 
During  early  planning,  test  planners  outline  the  scope  and 
structure  of  OT&E  activities  in  the  context  of  the 
operational  tasks  defined  in  the  strategy-to-task 
framework.  Although  OT&E  activities  are  generally 
concentrated  in  the  bottom  two  levels  of  Figure  4A.1,  test 
planners  must  be  prepared  to  link  their  test  back  to  any 
level  if  required.  To  do  this  effectively,  they  must 
comprehend  how  the  system  functions  within  the  entire 
hierarchical  framework.  Tests  that  are  not  well-linked 
will  lack  scientific  rigor,  operational  relevance,  and  be 
incapable  of  answering  senior  decision  makers’  questions. 


STRATEGY-TO-TASK  FRAMEWORK 


NATIONAL  SECURITY  STRATEGY 

NATIONAL  MILITARY  STRATEGY 

REGIONAL  (THEATER)  MILITARY  STRATEGY 
4^ 

REGIONAL  OPERATIONAL  OBJECTIVES 


OT&E 

FOCUS 


- ♦ - 

OPERATIONAL  TASKS 

4^ 

SYSTEM  FUNCTIONS  &  CHARACTERISTICS 


Figure  A4.1.  Strategy-to-Task  Framework, 
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5.  How  Linking  is  Done.  The  cost  and  operational 
effectiveness  analysis  (COEA)  provides  the  direct  link 
between  military  strategies,  tasks,  and  require-ments  in 
the  MNS  and  ORD,  and  the  determination  of  operational 
effectiveness  and  suitability  in  OT&E.  Users,  assisted  by 
OT&E  planners,  determine  in  the  COEA  what  it  takes  to 
make  a  difference  in  terms  of  engagement  or  battle  out¬ 
come  in  order  to  judge  between  competing  alternative 
systems.  The  measures  of  effectiveness  (MOE) 
developed  at  the  operational  task  level  (or  below)  in  the 
COEA  should  be  the  same  ones  used  in  the  test  concept 
and  test  plan. 

5.1.  A  well-  executed  COEA  includes  MOEs  reflecting 
operational  effectiveness  and  suitability  that  can  be 
objectively  tested.  An  MOE  is  a  measure  of  a  system’s 
performance  or  a  characteristic  that  indicates  the  degree  to 
which  it  (or  operational  task  element)  performs  an 
operational  task  or  meets  a  requirement  under  specified 
conditions.  MOEs  should  be  defined  to  enable  testers  to 
quantitatively  measure  the  degree  of  operational  task  (or 
mission)  accomplishment,  but  qualitative  measures  are 
also  useful,  MOEs  should  be  developed  to  a  level  of 
specificity  such  that  a  system’s  effectiveness  during 


testing  can  be  assessed  with  the  same  criteria  used  in  the 
COEA. 

5.2.  Some  MOEs  in  the  COEA  may  not  be  testable  in 
OT&E  because  they  are  measures  of  how  well  campaign- 
or  regional- level  strategies  are  accomplished.  In  these 
instances,  users  must  develop  more  specific  MOEs  at  the 
task  level  or  below  that  are  objectively  measurable  in  test. 
MOEs  are  used  singly  or  in  combination  in  the  test 
concept  and  test  plan  to  answer  the  COIs  which  support 
senior-level  decision  making. 

5.3.  Measures  of  performance  (MOP)  are  quantitative  (or 
qualitative)  measures  of  a  system’s  capabilities  or 
characteristics.  They  articulate  the  lowest  levels  of 
physical  performance  such  as  weight,  speed,  or  range. 
MOPs  should  relate  to  MOEs  such  that  the  effects  of  a 
change  in  MOP  can  be  related  to  a  change  in  the  MOE. 
Multiple  MOPs  may  be  aggregated  to  support  an  MOE  or 
critical  operational  issue  (COI),  just  as  multiple  MOEs 
may  be  aggregated  to  support  a  COI  or  operational  task. 

6.  How  Planning  is  Started.  Test  planners  must  first 
understand  the  system  in  terms  of  the  strategy-to-task 
framework  and  the  COIs  as  shown  in  Figure  A4.2. 
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TEST  CONCEPT  &  TEST  PLAN  DEVELOPMENT 


Figure  A4.2.  Test  Concept  and  Test  Plan  Development. 


6.1.  Critical  operational  issues  are  key  questions  about 
the  system  which  must  be  answered  for  senior  decision 
makers.  They  are  developed  by  the  operating  command 
with  AFOTEC  concurrence.  They  are  extracted  from  any 
number  of  documents  and  sources  and  help  test  planners 
and  decision  makers  focus  on  the  real  "show-stopping" 
concerns.  COIs  influence  the  overarching  thrust  of  the 
test  program  without  prescribing  any  of  the  structural 
details. 

6.2.  To  answer  the  COIs,  the  test  planner’s  focus  must 
shift  from  "What  the  system  is  required  to  do,"  to  "How 
do  we  measure  the  system’s  performance."  Therefore,  test 
planners  pursue  the  test  design  "problem"  from  a  different 
perspective  (the  horizontal  direction  in  Figure  A4.2)  than 


was  used  to  develop  the  COEA  in  the  strategy-to-task 
framework  (the  vertical  direction).  Test  planners  must 
organize  the  available  MOEs  and  MOPs  in  ways  that  best 
support  objective,  scientifically-based  test  and  evaluation 
of  the  system’s  ability  to  accomplish  the  operational  tasks. 
If  there  are  insufficient  MOEs  and  MOPs  to  answer  the 
COIs,  then  the  operating  command  and  testers  must 
develop  addition  measures  until  the  questions  can  be 
answered. 

7.  The  OT&E  Test  Concept,  The  test  concept  is  a  road 
map  driven  by  the  system’s  requirements  documents,  the 
acquisition  strategy,  and  the  basic  principles  of  OT&E.  It 
serves  three  basic  functions:  articulates  how  the  OT&E 
will  validate  system  performance  in  terms  of  strategy- to- 
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task  and  the  system  requirements;  describes  how  OT&E 
activities  support  the  system  acquisition  strategy;  and 
provides  the  rationale  for  all  subsequent  OT&E  planning. 
Test  planners  outline  how  they  will  use  available  test 
resources,  facilities,  ranges,  analysis  techniques,  and 
modeling  and  simulation  (M&S)  to  reduce  acquisition 
risk  and  answer  the  COIs.  Rough  estimates  are  made  of 
test  scenarios,  assumptions,  threat  laydowns,  schedules, 
resource  requirements,  and  test  limitations.  The  test 
concept  forms  the  basis  of  an  executable  OT&E  plan.  For 
OSD  oversight  programs,  OSD/DOT&E  requires  the  test 
agency  to  brief  their  test  concept  at  least  120  days  prior  to 
the  start  of  test.  HQ  USAF/TE  must  preview  and  approve 
all  test  concept  briefings  prior  to  presentation  to 
OSD/DOT&E. 

8.  Developing  the  OT&E  Plan.  The  OT&E  plan 
translates  the  test  concept  into  detailed  test  objectives, 
scenarios,  methods,  procedures,  resources,  and 
responsibilities.  Its  main  purpose  is  to  answer  the  COIs 
using  the  scientific  method  in  an  independent, 
operationally  realistic  investigation.  Initially  outlined 
during  early  planning,  the  OT&E  plan  is  fully  developed 
and  exercised  during  detailed  planning. 

8.1.  The  OT&E  plan  must  maintain  linkage  to  the 
military  need  and  the  MOEs  developed  for  the  system  in 
the  MNS,  ORD,  COEA,  and  the  strategy-to-task 
framework.  The  MOEs,  MOPs,  and  evaluation  criteria 
should  provide  good  indications  of  whether  the  system 
can  meet  user  requirements  in  realistic,  total  systems 
("end-to-end")  operational  scenarios, 

8.2.  Test  planners  and  analysts  break  down  operational 
tasks  into  manageable  test  objectives  for  which  specific 


data  is  collected.  They  devise  specialized  tools,  analysis 
techniques,  and  test  methodologies  to  evaluate  each 
operational  task  element.  A  variety  of  tools  may  be 
needed,  to  include  modeling  and  simulation.  Seldom  will 
a  single  test  objective,  model,  simulation,  or  evaluation 
technique  produce  all  the  necessary  data  for  an  MOE  or 
MOP.  Multiple  tools  and  methods  increase  confidence  in 
the  validity  of  the  test  results. 

8.3.  Occasionally  requirements,  thresholds,  or  criteria  are 
missing,  not  attainable,  not  "testable"  or  verifiable  during 
test,  or  need  further  clarification  or  analysis.  In  these 
cases,  test  planners  must  assist  the  operating  command  in 
clearly  defining  the  missing  elements  in  such  terms  that 
can  be  demonstrated  in  objective  test  and  evaluation.  Test 
planners  do  not  establish  requirements  or  criteria. 

8.4.  For  a  variety  of  reasons,  the  test  plan  or  test  concept 
may  differ  from  the  operational  environment  assumed  in 
the  COEA  and  the  ORD.  Possible  differences  include 
less  robust  threat  representations,  less  than  totally  realistic 
operational  test  scenarios,  safety  and  operating 
restrictions,  small  sample  sizes,  and  configuration 
differences  between  the  test  articles  and  the  system  as 
represented  in  the  COEA.  These  test  limitations  and  their 
impacts  on  test  outcomes  should  be  explained  in 
quantitative  terms  if  possible. 

8.5.  During  the  evaluation,  testers  aggregate  the 
conclusions  drawn  at  the  MOP  level  upward  to  answer 
sequentially  higher-level  questions  at  the  MOE,  COI,  or 
operational  task  levels  as  shown  in  Figure  A4.2. 
Reporting  procedures  are  described  so  timely  information 
is  provided  to  decision  authorities. 
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GLOSSARY  OF  TERMS 


Accreditation.  In  computer  modeling  and  simulation,  an 
official  determination  that  a  model  or  simulation  is 
acceptable  for  a  specific  purpose.  (Joint  Pub  1-02) 

Critical  Operational  Issue  (COI).  A  key  operational 
effectiveness  or  suitability  issue  that  must  be  examined  in 
OT&E  to  determine  the  system’s  capability  to  perform  the 
mission.  A  COI  is  normally  phrased  as  a  question  to  be 
answered  in  evaluating  a  system’s  operational 
effectiveness  and/or  suitability.  (DoDI  5000.2) 

Deficiency  Report  (DR).  A  report  to  provide  data 
pertaining  to  both  deficiencies  and  proposed 
enhancements  on  operational  and  logistics  supportability 
of  new  systems  or  equipment  during  all  phases  of  T&E. 
(TO  00-35D-54) 

Joint  Reliability  and  Maintainability  Evaluation  Team 
(JRMET).  Assists  in  collecting,  analyzing,  and 
categorizing  R&M  data  during  DT&E  and  lOT&E.  The 
JRMET  is  chaired  by  a  member  of  the  SPO  and  includes 
representatives  from  the  supporting  and  operating 
commands,  test  organization,  the  operational  test  agency, 
and  when  appropriate,  contractor  personnel  as  technical 
advisors. 

Measure  of  Effectiveness  (MOE).  A  measure  of  a 
system’s  task  accomplishment. 

Measure  of  Performance  (MOP).  A  qualitative  or 
quantitative  measure  of  a  system’s  capabilities  or 
characteristics.  It  indicates  the  degree  to  which  that 
capability  or  characteristic  performs  or  meets  the 
requirement  under  specified  conditions. 

Operational  Effectiveness.  The  overall  degree  of 
mission  accomplishment  of  a  system  when  used  by 
representative  personnel  in  the  environment  planned  or 
expected  (e.g.,  natural,  electronic,  threat,  etc.)  for 
operational  employment  of  the  system  considering 
organization,  doctrine,  tactics,  survivability,  vulnerability, 
and  threat  (including  countermeasures,  initial  nuclear 
weapons  effects,  nuclear,  biological,  and  chemical 
contamination  (NBCC)  threats).  (DoDI  5000.2) 

Operational  Reliability.  The  probability  that  an 
operationally  ready  system  will  react  as  required  to 
accomplish  its  intended  mission  or  function  as  planned, 
excluding  the  effects  of  enemy  action,  may  be  specified  as 
an  estimated  or  an  achieved  reliability.  (APR  80-5) 

Operational  Suitability.  The  degree  to  which  a  system 
can  be  placed  satisfactorily  in  field  use,  with 
consideration 


given  to  availability,  compatibility,  transportability, 
interoperability,  reliability,  wartime  usage  rates, 
maintainability,  safety,  human  factors,  manpower 
supportability,  logistics  supportability,  natural 
environmental  effects  and  impacts,  documentation,  and 
training  requirements.  (DoDI  5000.2) 

Operational  Task.  An  individual  military  operation 
accomplished  in  support  of  an  operational  objective. 

Realistic  Test  Environment.  The  conditions  during  test 
under  which  the  system  is  expected  to  be  operated  and 
maintained,  including  the  natural  weather  and  climatic 
conditions,  terrain  effects,  battlefield  disturbances,  and 
enemy  threat  conditions  which  best  represent  the 
operational  environment. 

Reliability.  The  ability  of  a  system  and  its  parts  to 
perform  its  mission  without  failure,  degradation,  or 
demand  on  the  support  system.  (DoDI  5000.2)  (AFM  11- 
1) 

Survivability.  The  capability  of  a  system  to  avoid  or 
withstand  man-made  hostile  environments  without 
suffering  an  abortive  impairment  of  its  ability  to 
accomplish  its  designated  mission.  (DoDI  5000.2) 
Capability  of  a  system  to  accomplish  its  mission  in  the 
face  of  an  unnatural  (man-made)  hostile,  scenario- 
dependent  environment.  Survivability  may  be  achieved 
by  avoidance,  hardness,  proliferation,  or  reconstitution  (or 
a  combination).  (AFM  11-1) 

Susceptibility.  The  degree  to  which  a  device,  equipment, 
or  weapon  system  is  open  to  effective  attack  due  to  one  or 
more  inherent  weakness.  A  function  of  operational 
tactics,  countermeasures,  probability  of  enemy  fielding 
threat,  etc.  Susceptibility  is  considered  a  subset  of 
survivability.  (DoDI  5(X)0.2) 

Sustainability.  The  ability  to  maintain  the  necessary 
level  and  duration  of  operational  activity  to  achieve 
military  objectives.  A  function  of  providing  for  and 
maintaining  those  levels  of  ready  forces,  materiel,  and 
consumables  necessary  to  support  military  effort.  (Joint 
Pub  1-02) 

Test.  Any  program  or  procedure  that  is  designed  to 
obtain,  verify,  or  provide  data  for  the  evaluation  of 
research  and  development  (other  than  laboratory 
experiments);  progress  in  accomplishing  development 
objectives;  or  performance  and  operational  capability  of 
systems,  subsystems,  components,  and  equipment  items. 
(DSMC  Glossary) 
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Test  and  Evaluation  (T&E) .  Process  by  which  a  system 
or  components  are  compared  against  requirements  and 
specifications  through  testing.  The  results  are  evaluated 
to  assess  progress  of  design,  performance,  supportability, 
etc.  The  term  "evaluation"  denotes  the  review  and 
analysis  of  data  produced  during  current  or  previous 
testing  and  data  obtained  from  test  conducted  by  other 
government  agencies  and  contractors,  from  operation  and 
commercial  experience,  or  combinations  thereof. 

Test  Director.  Directs  the  test  team  and  oversees 
execution  of  OT&E  according  to  the  test  plan.  Exercises 
overall  responsibility  for  achieving  plan  objectives. 


Validation.  In  computer  modeling  and  simulation,  the 
process  of  determining  the  degree  to  which  a  model  or 
simulation  is  an  accurate  representation  of  the  real  world 
from  the  prespective  of  the  intended  uses  of  the  model  or 
simulation.  (Joint  Pub  1-02). 

Verification.  In  computer  modeling  and  simulation,  the 
process  of  determining  that  a  model  or  simulation 
implementation  accurately  represents  the  developer’s 
conceptual  desription  and  specifications.  (Joint  Pub  1-02) 

Weapon  System.  A  combination  of  one  or  more 
weapons  with  all  related  equipment,  materials,  services, 
personnel,  and  means  of  delivery  and  deployment  (if 
applicable)  required  for  self-sufficiency.  (Joint  Pub  1-02) 
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ACAT 

ADM 

AETC 

AFAE 

AFMC 

AFI 

AFOTEC 

AFSARC 

B-LRIP 

CEP 

COEA 

COI 

COMSEC 

COTS 

CSAF 

CT 

CTEIP 

DAB 

DAC 

DAE 

DoD 

DR 

DT&E 

DTIC 

EEFI 

EIS 

EOA 

FOIA 

FOT&E 

FOUO 

FY 

HOI 

HUMINT 

lAW 

IMINT 

IOC 

lOT&E 

JT&E 

JITC 

LOA 

LOU 

LRIP 


ABBREVIATIONS 


Acquisition  Category 
acquisition  decision  memorandum 
Air  Education  and  Training  Command 
Air  Force  Acquisition  Executive 
Air  Force  Materiel  Command 
Air  Force  Instruction 

Air  Force  Operational  Test  and  Evaluation  Center 
Air  Force  System  Acquisition  Review  Council 

beyond  low-rate  initial  production 

critical  intelligence  parameter 

Cost  and  Operational  Effectiveness  Analysis 

critical  operational  issue 

communications  security 

commercial  off-the-shelf 

Chief  of  Staff  of  the  Air  Force 

contractor  test 

Central  Test  and  Evaluation  Investment  Program 

defense  acquisition  board 
Designated  Acquisition  Commander 
defense  acquisition  executive 
Department  of  Defense 
deficiency  report 
development  test  and  evaluation 
Defense  Technical  Information  Center 

essential  elements  of  friendly  information 
environmental  impact  statement 
early  operational  assessment 

Freedom  of  Information  Act 
follow-on  operational  test  and  evaluation 
for  official  use  only 
fiscal  year 

headquarters  operating  instruction 
human  intelligence 

in  accordance  with 
imagery  intelligence 
initial  operational  capability 
initial  operational  test  and  evaluation 

joint  test  and  evaluation 
Joint  Interoperability  Test  Center 

letter  of  agreement 
letter  of  understanding 
low  rate  initial  production 


MAJCOM 


major  command 
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MNS 

mission  need  statement 

MOA 

memorandum  of  agreement 

MOE 

measure  of  effectiveness 

MOP 

measure  of  performance 

MOU 

memorandum  of  understanding 

M&S 

modeling  and  simulation 

MS 

milestone 

NATO 

North  Atlantic  Treaty  Organization 

NDI 

nondevelopmental  item 

OA 

operational  assessment 

O&M 

operations  and  maintenance 

OPSEC 

operations  security 

ORD 

operational  requirements  document 

OSD 

Office  of  the  Secretary  of  Defense 

OTA 

operational  test  agency 

OT&E 

operational  test  and  evaluation 

OUE 

operational  utility  evaluation 

PE 

program  element 

PEO 

program  executive  officer 

PI 

program  introduction  document  (also  PID) 

PH) 

program  introduction  document  (also  PI) 

PMD 

program  management  directive 

QOT&E 

qualification  operational  test  and  evaluation 

R&D 

research  and  development 

RTO 

responsible  test  organization 

SAE 

Service  Acquisition  Executive 

SAF 

Secretary  of  the  Air  Force 

SECDEF 

Secretary  of  Defense 

SIGINT 

signals  intelligence 

SM 

single  manager 

SOW 

statement  of  work 

SPD 

system  program  director 

sss 

source  selection  sensitive 

STA 

system  threat  assessment 

STAR 

system  threat  assessment  report 

T&E 

test  and  evaluation 

TED 

threat  environment  description 

TEMP 

test  and  evaluation  master  plan 

TIPP 

test  investment  planning  and  programming 

TPWG 

test  planning  working  group 

TD&E 

tactics  development  and  evaluation 

UDS 

universal  documentation  system 

W&A 

verification,  validation,  and  accredidation 
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OT&E  CONTACTS 


Command  or 

Organization  Location 

HQ  USAF  1530  Air  Force  Pentagon, 

Rm  BE939 

Wash  DC  20330-1530 

1530  Air  Force  Pentagon, 

Rm  4D866 

Wash  DC  20330-1530 

HQ  AFOTEC  8500  Gibson  Blvd  S.E. 

Kirtland  AFB  NM  871 17-5558 
Plans,  Policy,  and 
Requirements  Directorate 
Policy  and  Procedures  Division 
Resource  Management  Directorate 
Research  Services  Directorate 
System  Analysis  Directorate 
Safety  Directorate 
Security  Directorate 
Special  Test  Directorate 
Weapon  Systems  Directorate 
C4I  Systems  Directorate 
Space  and  Missile 
Systems  Directorate 
Weather  Support  Directorate 


AFEWC 

Liaison  Officer  at  AFOTEC 


Office  dsn 

Symbol  Number 

TEP*  225-0900 

ter  227-1165 


XR 

246-5141 

XRX* 

246-5242 

RM 

246-0311 

RS 

246-5341 

SA 

246-5331 

SE 

246-5320 

SP 

246-5207 

ST 

246-0631 

TF 

246-5326 

TK 

246-5246 

TS 

246-4209 

WE 

246-2644 

OL-CH  246-8450 


AHC 

Operating  Location  at  AFOTEC 


OL-AF 


246-2361 


AFMC 


Liaison  Officer  at  AFOTEC 

HQ  AFOTEC/SA 

246-5450 

AFOTEC 

Det  1 

203  Losie  St 

Scott  AFB  IL  62225-5363 

CC 

576-3524 

Det2 

303  W.  C  Avenue,  Suite  102 

Eglin  AFB  FL  32542-6853 

CC 

872-8796 

Det  4 

4146  East  Bijou 

Colorado  Springs  CO  80909-6988 

CC 

692-6454 

Det  5 

30  North  Wolfe  Ave 

Edwards  AFB  CA  93523-5000 

CC 

527-3666 

OL-RC 

APO  AE  09094 

CC 

489-6721 
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OL-TF 

Nellis  AFB  NV  89191-5000 

CC 

682-2916 

OL-DL 

PO  Box  1 1037 

Tucson  AZ  85734-1037 

CC 

924-6918 

OL-PC 

730  Patrick  Ave 

Falcon  AFB  CO  80912-7300 

CC 

560-1110 

ACC 

Langley  AFB  VA  23665-5001 

DRM* 

574-7597 

AHA 

Kelly  AFB  TX  78243-5000 

XP* 

969-2157 

AFMC 

Wright-Patterson  AFB  OH 

45433-5001 

(TBD) 

(TBD) 

CSTI 

Wright-Patterson  AFB  OH 

45433-5001  LAT* 

785-3161 

AFRES 

Robins  AFB  GA  31098-6001 

XPQ* 

468-5400 

AFSOC 

Hurlburt  Fid  FL  32544-5000 

XPT* 

SMOTEC 

Hurlburt  Fid  FL  32544-5000 

CC 

AFSPACECOM  Peterson  AFB  CO  80914-5001 

DOT* 

692-9643 

AMC 

Scott  AFB  IL  62225-5001 

XRT* 

576-3903 

USAFMC 

Pope  AFB  NC  28308-5000 

CC 

486-2443 

ATC 

Randolph  AFB  TX78 150-5001 

XPR* 

487-3092 

AWS 

Scott  AFB  IL  62225-5008 

PMA* 

576-3268 

NGB 

Pentagon  Wash  DC  20310-2500 

AQ* 

224-0188 

AATC 

Box  11037,  Tucson  AZ  85734-1037 

DT 

924-6927 

PACAF 

Hickam  AFB  HI  96853-5001 

DOQQ* 

449-6331 

USAFE 

APO  NY  09012-5001  DOQ* 

480-6921 

^Command  Focal  Point  for  OT&E 


